Well, first you need to decide if it's worth caring about that, because in order to deal with it you have to impose restrictions beyond redistribution, which means your license is no longer a pure copyright license, which arguably means it violates Freedom Zero and cannot qualify as a Free Software license per the FSF's own definition (of course they say the AGPL does, as it's their own license, but by a strict reading, it doesn't; by a looser reading it does, but it is also toothless). (It would qualify as open source under the OSI definition, but the OSI definition is unfortunately really poor in this regard, as it does not require lack of usage restrictions; you could say "you can't run my app on more than one CPU core or have more than 4 simultaneous users" and that license would still meet the OSI guidelines!)
Now, if you're prepared to make that concession... I don't know what to tell you. The AGPL is a terrible option as I've mentioned. However, the FSF deliberately made the GPLv3 and AGPLv3 an interoperable pair, so if you want to link with GPLed software, you have no other option. If you don't have any GPLed dependencies, you could write your own license, and properly spell out what "remote network interaction" means for your app (and especially how transitive it is to being integrated into other systems, interaction with proxies, frontends, etc - all of which the AGPL is completely mute on), and how the source should be distributed. Ideally you'd build in a source code offer mechanism into your network protocol and make the app able to serve its own source code, so as to not burden your users with licensing traps they have to take explicit action on, and then make the licensing conditional on that.
In principle, since you can always make a license less strict via exceptions, if you're willing to accept that the AGPL may be completely ineffective at closing the loophole depending on interpretation, then you could use the AGPL and spell out what the required extent of the network interaction requirements is in the form of an exception; this would lessen the problematic burden of the AGPL on your users, but it doesn't buy you any extra protections, as you can only go weaker, not stronger.
