Why language models hallucinate • Hacker News Board

Re: Why language models hallucinate

#231

Post by mock-possum » Mon, Sep 08, 2025, 2:57 PM UTC

CGMthrowaway wrote: post #215

aleph_minus_one wrote: post #2

> Think about it like a multiple-choice test. If you do not know the answer but take a wild guess, you might get lucky and be right. Leaving it blank guarantees a zero. In the same way, when models are graded only on accuracy, the percentage of questions they get exactly right, they are encouraged to guess rather than say “I don’t know.” To me, this seems to be an "US-American" way of thinking about multiple-choice t…

>> Think about it like a multiple-choice test. If you do not know the answer but take a wild guess, you might get lucky and be right. Leaving it blank guarantees a zero. In the same way, when models are graded only on accuracy, the percentage of questions they get exactly right, they are encouraged to guess rather than say “I don’t know.” For TIMED multiple-choice tests (and the timed constraint makes sense in OP ana…

Yeah when you realize that the fake answers to the test have been created by humans, you can predict what false answers might look like - you can even get a feel for the kind of false answers that the test’s author tends to provide, and by the end of the test you can start to spot them fairly confidently. You still check your work, obviously, but it’s like picking up on a poker player’s tell - it gives you an edge.

Re: Why language models hallucinate

#232

Post by cindyllm » Mon, Sep 08, 2025, 3:13 PM UTC

mock-possum wrote: post #231

CGMthrowaway wrote: post #215

Earlier quoted context omitted.

>> Think about it like a multiple-choice test. If you do not know the answer but take a wild guess, you might get lucky and be right. Leaving it blank guarantees a zero. In the same way, when models are graded only on accuracy, the percentage of questions they get exactly right, they are encouraged to guess rather than say “I don’t know.” For TIMED multiple-choice tests (and the timed constraint makes sense in OP ana…

Yeah when you realize that the fake answers to the test have been created by humans, you can predict what false answers might look like - you can even get a feel for the kind of false answers that the test’s author tends to provide, and by the end of the test you can start to spot them fairly confidently. You still check your work, obviously, but it’s like picking up on a poker player’s tell - it gives you an edge.

[dead]

Re: Why language models hallucinate

#233

Post by yreg » Mon, Sep 08, 2025, 3:52 PM UTC

kelnos wrote: post #201

yreg wrote: post #200

Maybe it goes against the definition but I like saying that _all_ output is a hallucination, when explaining LLMs. It just happens that a lot of that output is useful/corresponding with the real world.

Well yes, it goes against the accepted definition. And if all output is hallucination, then it's not really a useful way to describe anything, so why bother?

I find it useful to underline the intrinsic properties of LLMs. When an LLM makes up something untrue, it's not a 'bug'.

I think that thinking of all LLM output as 'hallucinations' while making use of the fact that these hallucinations are often true for the real world is a good mindset, especially for nontechnical people, who might otherwise not realise.

Re: Why language models hallucinate

#234

Post by zbentley » Mon, Sep 08, 2025, 3:52 PM UTC

riwsky wrote: post #187

zbentley wrote: post #145

Earlier quoted context omitted.

I think your assessment of the academic take on AI is wrong. We have a rather thorough understanding of the how/why of the mechanisms of LLMs, even if after training their results sometimes surprise us. Additionally, there is a very large body of academic research that digs into how LLMs seem to understand concepts and truths and, sure enough, examples of us making point edits to models to change the “facts” that the…

Here’s where you're clearly wrong. The correct favorite in that corpus is Golden Gate Claude: https://www.anthropic.com/news/golden-gate-claude

Both are very good! I usually default to sharing the Bau Lab's work on this subject rather than Anthropic's because a) it's a little less fraught when sharing with folks who are skeptical of commercial AI companies, and b) because Bau's linked research/notebooks/demos/graphics are a lot more accessible to different points on the spectrum between "machine learning academic researcher" and "casual reader"; "Scaling/Towards Monosemanticity" are both massive and, depending on the section, written for pretty extreme ends of the layperson/researcher spectrum.

The Anthropic papers also cover a lot more subjects (e.g. feature splitting, discussion on use in model moderation, activation penalties) than Bau Lab's, as well--which is great, but maybe not when shared as a targeted intro to interpretability/model editing.

Re: Why language models hallucinate

#235

Post by drudolph914 » Mon, Sep 08, 2025, 4:48 PM UTC

I am an educator alongside being an engineer, so I've had to think about how to explain this topic to people in ways that give them some kind of intuition/insight. I don't have a good take for non-stem people, but I think I have a better explanation for people who are CS adjacent

I like to explain this whole hallucination problem by stating that LLMs are 2 different machines working together. one half of the machine is all the knowledge it was trained on, and you can think of this knowledge as an enormous classic tree you learn in CS classes; and each node in this tree is a token. the other half of the machine is a program that walks through this enormous tree and prints the token it's on

when you think of it like this, 3 things become immediately obvious

1. LLMs are a totally deterministic machine

2. you can make them seem smart by randomizing the walk through the knowledge tree

3. hallucinations are a side effect of trying to randomize the knowledge tree walk

I find it interesting that LLM companies are trying to fix such a fundamental problem by training the model to always guess the correct path. the problem I see with this approach is that 2 people can enter the same input text, but want 2 different outputs. if there isn't always a _correct path_ then you can't really fix the problem.

the only 2 options you have to “improve” things is prune and or add better data to the knowledge tree, or you’re trying the make the program that walks the knowledge tree take better paths.

the prune/add data approach is slightly better because it’s improving the quality of the token output. but the downside is you quickly realize that you need a fire hose of new human data to keep improving - but much of the data out there is starting to be generated by the LLMs - which leads to this inbreeding effect where the model gets worse

the 2nd approach feels less ideal because it will slow down the process of generating tokens.

all of this to say, from this point on, it’s just hacks, ducktape, and bandaids

Re: Why language models hallucinate

#236

Post by sfink » Mon, Sep 08, 2025, 6:13 PM UTC

I have mixed feelings about AI, but love the posts and papers that dig into how they work. Except, as this post shows, I seem to vastly prefer Anthropic's posts to OpenAI's.

> Claim: Hallucinations are inevitable.

> Finding: They are not, because language models can abstain when uncertain.

Please go back to your marketing cave. "Claim: You'll get wet if it rains. Finding: You will not, because you can check the weather report and get inside before it starts raining."

Sure, language models could abstain when uncertain. That would remove some hallucinations [a word which here means, make statements that are factually untrue. Never mind that that's often what we want them to do.] Or when certain about something that their training data is flawed or incomplete about. Or when certain about something but introspection shows that the chain of activations goes through territory that often produces hallucinations. Or when certain about something that is subjective.

"Uncertainty" is a loaded term; these things don't think in the way that the definition of the word "certain" is based on, since it's based on human thought. But that aside, LLM uncertainty is very obviously a promising signal to take into account, and it's interesting to see what costs and benefits that has. But eliminating one cause does not prove that there are no other causes, nor does it address the collateral damage.

"Write me a story about Bill."

"I'm sorry Dave, Bill is hypothetical and everything I could say about him would be a hallucination."

"Write a comment for the function `add(a, b) = a + b`."

"// This function takes two numbers and adds them toget... I'm sorry Dave, I don't know how many bits these numbers are, what the behavior on overflow is, or whether to include the results of extreme voltage fluctuations. As a result, I can't produce a comment that would be true in all circumstances and therefore any comment I write could be construed as a hallucination."

Re: Why language models hallucinate

#237

Post by p_v_doom » Tue, Sep 09, 2025, 7:05 AM UTC

freehorse wrote: post #72

fumeux_fume wrote: post #28

I like that OpenAI is drawing a clear line on what “hallucination” means, giving examples, and showing practical steps for addressing them. The post isn’t groundbreaking, but it helps set the tone for how we talk about hallucinations. What bothers me about the hot takes is the claim that “all models do is hallucinate.” That collapses the distinction entirely. Yes, models are just predicting the next token—but that do…

> What bothers me about the hot takes is the claim that “all models do is hallucinate.” That collapses the distinction entirely That is a problem for "Open"AI because they want to sell their products, and because they want to claim that LLMs will scale to superintelligence. Not for others. "Bad" hallucinations come in different forms, and what the article describes is one of them. Not all of them come from complete u…

The problem is not so much IMO that all models hallucinate. Its more that our entire reality, especially as expressed through the training data - text, is entirely constructed. There is no difference in the world made by the text, say when it comes to the reality of Abraham Lincoln and Bilbo Baggins. We often talk about the later as if he is just as real. Is Jesus real? Is Jesus god? Is it hallucination to claim the one you dont agree with? We cant even agree amongst oursevles what is real and what is not.

What we perceive as "not hallucination" is merely a very big consensus supported by education, culture, personal beliefs and varies quite a bit. And little in the existence of the model gives it the tools to make those distinctions. Quite the opposite

Re: Why language models hallucinate

#238

Post by red75prime » Tue, Sep 09, 2025, 9:55 AM UTC

sirwhinesalot wrote: post #198

red75prime wrote: post #185

Earlier quoted context omitted.

> And that problem is one that the AI field has already spent 60 years on I hope you don't think that the solutions will be a closed-form expression. The solution should involve exploration and learning. The things that LLMs are instrumental in, you know.

Not the same person but I think the "structure" of what the ML model is learning can have a substantial impact, specially if it then builds on that to produce further output. Learning to guess the next token is very different from learning to map text to a hypervector representing a graph of concepts. This can be witnessed in image classification tasks involving overlapping objects where the output must describe thei…

I mostly agree. But, next token prediction is a pretraining phase of an LLM, not all there is to LLMs.

Re: Why language models hallucinate

#239

Post by parentheses » Tue, Sep 09, 2025, 4:32 PM UTC

fumeux_fume wrote: post #28

I like that OpenAI is drawing a clear line on what “hallucination” means, giving examples, and showing practical steps for addressing them. The post isn’t groundbreaking, but it helps set the tone for how we talk about hallucinations. What bothers me about the hot takes is the claim that “all models do is hallucinate.” That collapses the distinction entirely. Yes, models are just predicting the next token—but that do…

Yes. Maybe a better way to put it would be, "all models guess every time because they are stochastic in nature. However, we only want the answers with high confidence."

Re: Why language models hallucinate

#240

Post by parentheses » Tue, Sep 09, 2025, 4:33 PM UTC

I think a large issue at play here is post training. Pre training models the original distribution of input data. RL techniques tweak the models to "behave". This step changes how the models "think" in a fundamental way .