I think AI hallucinations are more than just incorrect information. They're incorrect information coupled with an air of certitude. Sometimes with a whole backstory. And often oblivious to any inherent contradictions. It's like, if you ask a random person what years Lyndon B Johnson was president, they might say something like 1964-1968, shrugging with a bit of uncertainty. And both of those years would be incorrect but close. Whereas an AI (if hallucinating) would say something like Lyndon B Johnson was president from April 15, 1865 – March 4, 1869, after the shooting of John F. Kennedy in Ford's theater. The dates are precise but completely wrong, as are the accompanying details. Then if you ask the AI when John F. Kennedy died, it might correctly respond November 22, 1963, in Dallas, TX, totally unaware how this information doesn't match with the erroneous information it gave earlier.
I've been thinking there's some parallels between how AIs hallucinate and how human toddlers do. If you ask a toddler/young child a question about a fact they don't know, they will usually say, "iuno" (even when they should), but depending on the child and the circumstances, they will sometimes just make up a story on the spot and sound as if they believe it. "Who invented ice cream?" "Santa Claus! Mommy left him milk and cookies and he turned it into ice cream." It doesn't make any real sense but it seems facially plausible in their universe.
But somewhere between first learning to speak and around 7ish, kids become markedly more accurate how they model the world, and their responses become correspondingly less fanciful. And they continue to improve beyond that point.
So how are kids doing what LLMs are currently incapable of? How do we teach ourselves not to hallucinate? Or do we, really? I mean, if I tell myself I'm going to make it through the intersection before the light turns red, but I end up running the red light, was I just mistaken, or was that a self-delusion, i.e., a mini-hallucination of sorts? Probably a self-driving car would be less likely to make that category of mistake, so maybe I shouldn't be so smug about being grounded in reality.