You can say that, but then I can say why do we care about whether something is a productive use of capital? What I care about is life being good for human beings, not "productive use of capital".

We care because productive use of capital create income that is taxed, that pays for things like healthcare, social services, etc.

Could we live with less productive capital? Sure. It wouldn't be anywhere near as nice.