I think AI doomers are mostly dumb, but this argument is not very good. The Halting Problem is a technical fact: you can't _prove_ that a program terminates in general. This doesn't mean that all programs are ambiguous with respect to termination. In fact, with appropriate limitations imposed, many programs could be proven to terminate. And even without limitations humans wrangle code all the time. If the presumption that a machine intelligence is at least as smart as a person, I don't see why they'd be any more likely than we are to run into non-terminating programs or whatever.
Furthermore, it doesn't matter for both practical and philosophical reasons. Practically, I don't see why a super-intelligence would just shut itself down to run some new code without various kinds of testing procedures. Also, there aren't any Turing machines anyway, since they need infinite memory, which even a super-intelligence doesn't have. I don't really see how the halting problem is a material problem for a super smart agent.
It's not just crashing/halting. See Rice's Theorem. The machine can't predict its own future behavior in most ways that are important.
I'm not an expert in this stuff, and my point was that a serious treatment of super-intelligence should address such limitations to computations. In particular the "super" part seems to imply solving exponential-complexity problems in linear time. I remember looking in NB's book and not finding it.
I don't see why that is the case. I'm much more intelligent than a racoon and I have yet to grapple with Rice's Theorem. There is no reason I can think of to believe that Rice's Theorem is a serious constraint on intelligence beyond my own. In general, such an agent isn't particularly interested in proving facts about its own program (at least I can't see why it would care any more than we are interested in proving mathematical properties of our own brains). It is interested in maximizing some objective function, which can transparently be done without thinking much at all about Rice's Theorem (all systems which train neural networks and indeed, even simpler optimization problems, pursue such maximization with nary a thought towards Turing or Rice's Theorems).
Furthermore, it doesn't matter for both practical and philosophical reasons. Practically, I don't see why a super-intelligence would just shut itself down to run some new code without various kinds of testing procedures. Also, there aren't any Turing machines anyway, since they need infinite memory, which even a super-intelligence doesn't have. I don't really see how the halting problem is a material problem for a super smart agent.