> Whether you consider this architecture bananas or not depends on your security model.

Knowing about the security risks is good. In all of my use cases I control both client and server machine. The remote SSH capability just lets me use the UI comfortably on my laptop.