Hallucination is inevitable: An innate limitation of large language models

Tools are not people and people should not be considered as tools. Imagine your hammer only hitting the nail 60% of the time! But workers should be allowed to stop working to negotiate work conditions.

Yeah good point. But dreams are easily distinguishable from reality.

Religion is often indistinguishable from truth and reality to those who hallucinate it.

> Notice that you had to notice the error and had to prompt it to double check. Lots of complicated things going on here. Many (most?) humans will fail somewhere along this trajectory.

Sure. This was covered by my statement above: "These algorithms are incomplete and buggy".

> “It was able to reason intelligently” imbues qualities that I am skeptical is reasonable to attribute to this very narrow domain - what’s an example where it showed intelligent reasoning capabilities?

Here's an example. I asked it to analyze the case where we always pick option "b" in my example above. It took my word problem and boiled it down to an equation:

  ChatGPT: To find the threshold where the behavior switches from decreasing to increasing when always choosing option "b", we need to identify the smallest integer starting value that, after being doubled and reduced by 40, leads to an equal or larger integer in the next iteration. We're looking for the smallest integer x where 2x - 40 >= x 

This was part of a longer conversation where it analyzed different properties and outcomes of the "game rules" that I gave it.

As you pointed out, it got some things wrong and had to be corrected. But Socratic reasoning works fairly well to guide it. It can find errors in its own reasoning. For instance, if asked to actually calculate a few iterations for a given case, it will find its own errors in its claims about that case.

Is it useful right now? Maybe, maybe not, depends on your use case. It definitely takes a lot of thinking on your own and guiding it. At some points it goes from seemingly intelligent to downright pigheaded and stupid.

But in my view there is absolutely no way a lookup table algorithm can contain enough data to be anywhere near the level of responses we're seeing here.

> you can train the network to act as if it knows its knowledge limits

Humans need more training to do the same.

And this does not answer the question of whether there's self-reflection going on. Practical LLMs available today are perfectly capable of telling you about their own reasoning process.

And much like a human, an LLM is incapable of fully explaining it's reasoning process from first principles to a human. Why is that? Probably because humans are too limited to grok large scale complex processes like the human brain or an LLM.

Finally, there is no difference between acting as if you know your training limits and knowing your training limits, if your acting is good enough. This goes for both humans and LLMs.

This is sort of like the compression algorithm "problem." For the overwhelming majority of inputs, compression algorithms don't compress, and it can be proven that on average they don't work. But we're not really interested in compressing things on average. What we use compression for amounts to edge cases of highly regularized or repeatable data.

Thus the fact that LLMs can be proven in general to hallucinate doesn't necessarily imply that they must hallucinate in the types of situations for which we use them for. The paper itself discusses a number of mitigating strategies -- such as supplementing their training data with current information or using multiple LLMs to vote on the accuracy of the outcome -- only to basically brush them aside and advise not to use LLMs in any sort of critical situation. And that's probably true enough today, but in the future I think these strategies will greatly reduce the severity of these hallucinations. Just as we as human beings have developed strategies to reduce our reliance on pure memory.

This reminds me of a deposition I had to give a number of years back. One of the lawyers asked me if I remembered how the plaintiff and I came to discuss a certain accusation leveled at him by the defendant. And I confidently stated, "Sure, he and I used to have various conversations about the issue and one day he the plaintiff brought up this thing that defendant said to him." And the lawyer says, if you want to, you can refer to your phone text log to refresh your memory. Then I looked at my phone, and the truth was that I myself had spoken to the defendant, and she told me the accusation, and then I went and shared it with the plaintiff. So, I basically remembered the situation exactly backwards, i.e., a hallucination, which I was able to repair by referring to real world information instead of just my memory.