You're assumption that it'd move server side is incorrect. It would simply disappear. When you use a javascript library the browser has to process and load all the functions in the library, even the functions that aren't used. This wastes memory and processing cycles on the client computer.
> When you use a javascript library the browser has to process and load all the functions in the library.
That simply isn't true. Your device does load a library (usually minified), and parses the JS to be sure it is runnable. But it doesn't run every function just because it is loaded in memory. Even if it did, it happens on app load, once... then runs functions on-demand. And the size of the load can be mitigated with appropriate use of tree-shakable libraries. But without JS, the server regenerates an entire page, every time you do something, which takes some resources. It sends it over the wires, which takes some small amount of resources. The mobile device loads an entire page again.
I do agree there is wasted processing in both scenarios. But that waste doesn't just disappear when you do server-side work.