Exactly right, they do not have a concept of true and false as unsupervised learning simply makes them good at predicting the next token. But I think there is an over-confidence bias in the training data sample. On top of that, instruction tuning wants definitive answers, as you say. And finally, RLHF probably favors over-confident answers because people like that. From start to finish, over-confidence bias is everywhere — we both produce over-confident training data, and tune for over-confident answers.

Or, well... that's what I think. See, I've not trained an LLM, I have only read about it online, and very little in books I have on the topic. I did some machine learning exercises in university, and that's the extent of my practical knowledge. And as I say that, the impact of my words goes down, right? They are taken less seriously than if someone said all that stuff about LLMs but never said they don't have practical experience. And yet, this makes the information as it is presented more exact, the limitations are clear, so it is more useful.

More useful, but far less appealing... This is a really interesting topic.