Why language models hallucinate • Hacker News Board

Re: Why language models hallucinate

#111

Post by didibus » Sat, Sep 06, 2025, 9:26 PM UTC

amelius wrote: post #9

They hallucinate because it's an ill-defined problem with two conflicting usecases: 1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original. 2. If I ask it a question, I want it to reply with facts. It should not make up stuff. LMs were originally designed for (1) because researchers thought that…

The word "hallucination" mis-characterizes it.

LLMs predict the likely tokens to follow the context. And they can make incorrect predictions.

LLMs therefore don't have perfect accuracy of prediction. When their predictions are incorrect, people say they "hallucinate".

Nobody questions why predictive weather models aren't perfectly accurate, because it makes sense that a prediction can be wrong.

Marketing and hype has tried to sell LLMs as "logical rational thinkers" equal to human thinking. A human doing actual thinking knows when they are making stuff up. So if a human truly believes obviously false things to be true, it tends to be because they are hallucinating. Their thinking isn't wrong, they've lost track of reality to ground their thinking.

We've anthropomorphized LLMs to the point we wonder why are they hallucinating like we can offer a diagnostic. But if you stop anthropomorphising them and go back to their actual nature as a predictive model, then it's not even a surprising outcome that predictions can turn out to be wrong.

Re: Why language models hallucinate

#112

Post by jrm4 » Sat, Sep 06, 2025, 9:26 PM UTC

Yeah, no, count me in with those who think that "All they do is hallucinate" is the correct way to say this and anything else dangerously obscures things.

More than anything, we need transparency on how these things work. For us and for the general public.

"Hallucination" introduces the dangerous idea that "them getting things wrong" is something like a "curable disease" and not "garbage in garbage out."

No. This is as stupid as saying Google telling me a restaurant is open when it's closed is a "hallucination." Stop personifying these things.

Re: Why language models hallucinate

#113

Post by throwawaymaths » Sat, Sep 06, 2025, 9:27 PM UTC

RugnirViking wrote: post #81

throwawaymaths wrote: post #69

Earlier quoted context omitted.

you're missing the point. SAT multiple choice negatives for random guesses, fine, you could trivially use this sort of a strategy for assigning cost functions to a classifier and backpropagate. how do you give negative weight to a wrong answer when training a transformer?

isn't this just related to the question "how do you train a transformer"? you give it wrong examples, and use optimization algorithms to move away from that kind of completions

thats quite hard for the reasons i explained. might be solvable using q learning techniques, but those are not easy in the context of transformers iiuc

Re: Why language models hallucinate

#114

Post by throwawaymaths » Sat, Sep 06, 2025, 9:28 PM UTC

johnnyanmac wrote: post #79

throwawaymaths wrote: post #70

Earlier quoted context omitted.

fine. which part is the problem?

The part where it can't admit situations where there's not enough data/training to admit it doesn't know. I'm a bit surprised no one talks about this factor. It's like talking to a giant narcissist who can Google really fast but not understand what it reads. The ability to admit ignorance is a major factor of credibility, because none of us know everything all at once.

yeah sorry i mean which part of the architecture. "working as designed"

Re: Why language models hallucinate

#115

Post by [deleted] » Sat, Sep 06, 2025, 9:30 PM UTC

ACCount37 wrote: post #73

throwawaymaths wrote: post #69

Earlier quoted context omitted.

you're missing the point. SAT multiple choice negatives for random guesses, fine, you could trivially use this sort of a strategy for assigning cost functions to a classifier and backpropagate. how do you give negative weight to a wrong answer when training a transformer?

In RLVR? Quite easily. And OpenAI has induced hallucinations in o3 with RLVR mistakes, not with a failed pre-training run. They used o4-mini as an example - similar training to o3 and similar issues. Conversely, they have also designed a post-training system that has successfully reduced hallucinations in GPT-5.

[deleted]

Re: Why language models hallucinate

#116

Post by utyop22 » Sat, Sep 06, 2025, 9:32 PM UTC

rhubarbtree wrote: post #105

I find this rather oddly phrased. LLMs hallucinate because they are language models. They are stochastic models of language. They model language, not truth. If the “truthy” responses are common in their training set for a given prompt, you might be more likely to get something useful as output. Feels like we fell into that idea and said - ok this is useful as an information retrieval tool. And now we use RL to reinfo…

The reality is, language itself does not capture the entirety of what is really going on. And I'd get argue its the poorest way of expressing - but one that enables transmission through various mediums efficiently on a cost basis.

E.g. when I explain a concept, what comes to my mind is not a string of letters and words. There is a mix of imagery and even sounds that I may have acquired from learning about a concept - then I translate that into text so it can be communicated.

Theres a reason why people use native subtitles when watching netflix - text complements imagery and sounds.

Re: Why language models hallucinate

#117

Post by rhubarbtree » Sat, Sep 06, 2025, 9:32 PM UTC

ComplexSystems wrote: post #109

rhubarbtree wrote: post #105

I find this rather oddly phrased. LLMs hallucinate because they are language models. They are stochastic models of language. They model language, not truth. If the “truthy” responses are common in their training set for a given prompt, you might be more likely to get something useful as output. Feels like we fell into that idea and said - ok this is useful as an information retrieval tool. And now we use RL to reinfo…

> Trying to eliminate cases where a stochastic model the size of an LLM gives “undesirable” or “untrue” responses seems rather odd. Why? It seems no less odd than eliminating cases where it gives "undesirable" code snippets with hallucinated errors. This is very important and not odd at all.

To clarify, because you will be left with a biased language model. It will continue to hallucinate, and as you squeeze some hallucinations in one part of the language space you may well create new ones elsewhere. It doesn’t seem a solid line of attack

Re: Why language models hallucinate

#118

Post by ospray » Sat, Sep 06, 2025, 9:33 PM UTC

cainxinth wrote: post #110

I find the leader board argument a little strange. All their enterprise clients are clamoring for more reliability from them. If they could train a model that conceded ignorance instead of guessing and thus avoid hallucinations, why aren't they doing that? Because of leader board optics?

I think they are trying to communicate that their benchmarks will go down as they try to tackle hallucinations. Honestly I am surprised they didn't just say we think all benchmarks need a incorrect vs abstinence ratio so our cautious honest model can do well on that. Although they did seem to hint that's what they want.

Re: Why language models hallucinate

#119

Post by robertclaus » Sat, Sep 06, 2025, 9:35 PM UTC

While I get the academic perspective of sharing these insights, this article comes across as corporate justifying/complaining that their model's score is lower than it should be on the leaderboards... by saying the leaderboards are wrong.

Or an even darker take is that its coorporate saying they won't prioritize eliminating hallucinations until the leaderboards reward it.

Re: Why language models hallucinate

#120

Post by throwawaymaths » Sat, Sep 06, 2025, 9:36 PM UTC

ComplexSystems wrote: post #107

saghm wrote: post #76

Earlier quoted context omitted.

> so if you ask, "what is the capital of colorado" and it answers "denver" calling it a Hallucination is nihilistic nonsense that paves over actually stopping to try and understand important dynamics happening in the llm matrices On the other hand, calling it anything other than a hallucination misrepresents the idea of truth as being something that these models have any ability to differentiate between their outputs…

It isn't a hallucination because that isn't how the term is defined. The term "hallucination" refers, very specifically, to "plausible but false statements generated by language models." At the end of the day, the goal is to train models that are able to differentiate between true and false statements, at least to a much better degree than they can now, and the linked article seems to have some very interesting sugge…

your point is good and taken but i would amend slightly -- i dont think that "absolute truth" is itself a goal, but rather "how aware is it that it doesn't know something". this negative space is frustratingly hard to capture in the llm architecture (though almost certainly there are signs -- if you had direct access to the logits array, for example)