The argument is that you're running code on the remote host, and it could be compromised. The same argument can be made about any code you run on the remote.

VSCode may be seen as a larger attack vector due to its popularity; but maybe not as many won't use the SSH agent? It's also fairly common sense that you should never run it to mount on a production resource; but again, you shouldn't be able to ssh into a production machine anyway.