The difference with self-driving cars hype is that they need to be 99.999% good so pretty much perfect to be useful on road and be incorporated mainstream. AI doing some tasks 90% as good as human is good enough. Self driving cars got massively improved in the last 15 years. I remember 15 years ago when DARPA were doing their first self driving challenge and the current tech we have is like magic comparing to what we had back then.

But with software doesn't technical debt accumulate over time when low quality engineers keep working on it?

That's why starting projects with something like Cursor makes you seem superhuman, but as the project grows the AI is more likely to get stuck because of previous low-quality choices. Just like with driving cars, it seems like you need strong supervision. (at least for the foreseeable future)

It depends on what the 10% is. For example, if your code only works 90% of the time that's not exactly good enough to use...

need a few more 9's than that otherwise everyone would be dead within 50 years

99.999% on an annual basis is ~10x safer than human drivers