Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why language models hallucinate

openai.com

Post reply on HN
Print view View original

71–80 of 242 posts

Re: Why language models hallucinate

#71

Post by Jensson »

ACCount37 wrote: post #60
pessimizer wrote: post #47

Earlier quoted context omitted.

Do they "recognize" that they don't know the word, or are there just no statistically plausible surroundings that they can embed a nonsense word into other than settings that usually surround un-tokenizable words? If you told them to write a Lewis Carroll poem about a nonsense word, it wouldn't have any problem. Not because it "recognizes" the word as being like a nonsense word in a Lewis Carroll poem, but because th…

I don't think there's a direct link to the tokenizer - it's a higher level capability. You can stitch together a nonsense word out of common "word fragment" tokens and see if that impairs the LLM's ability to recognize the word as nonsense.

That is wrong, I just generated 5 random letters in python and sent it to gpt-5 and it totally failed to answer properly, said "Got it, whats up :)" even though what I wrote isn't recognizable at all.

The "capability" you see is for the LLM to recognize its a human typed random string since human typed random strings are not very random. If you send it an actual random word then it typically fails.

Re: Why language models hallucinate

#72

Post by freehorse »

fumeux_fume wrote: post #28

I like that OpenAI is drawing a clear line on what “hallucination” means, giving examples, and showing practical steps for addressing them. The post isn’t groundbreaking, but it helps set the tone for how we talk about hallucinations. What bothers me about the hot takes is the claim that “all models do is hallucinate.” That collapses the distinction entirely. Yes, models are just predicting the next token—but that do…

> What bothers me about the hot takes is the claim that “all models do is hallucinate.” That collapses the distinction entirely

That is a problem for "Open"AI because they want to sell their products, and because they want to claim that LLMs will scale to superintelligence. Not for others.

"Bad" hallucinations come in different forms, and what the article describes is one of them. Not all of them come from complete uncertainty. There are also the cases where the LLM is hallucinating functions in a library, or they reverse cause and effect when summarising a complex article. Stuff like this still happen all the time, even with SOTA models. They do not happen because the model is bad with uncertainty, they have nothing to do with knowledge uncertainty. Esp stuff like producing statements that misinterpret causal relationships within text, imo, reveals exactly the limits of the architectural approach.

Re: Why language models hallucinate

#73

Post by ACCount37 »

throwawaymaths wrote: post #69
ACCount37 wrote: post #42

Earlier quoted context omitted.

Yes, it's not nearly as easy as "just fix the evals". But better evals are still helpful, because they reward LLM vendors for trying to do the very-hard-to-do thing. Instead of rewarding them for training an LLM that's really good at emitting 7% confidence guesses.

you're missing the point. SAT multiple choice negatives for random guesses, fine, you could trivially use this sort of a strategy for assigning cost functions to a classifier and backpropagate. how do you give negative weight to a wrong answer when training a transformer?

In RLVR? Quite easily.

And OpenAI has induced hallucinations in o3 with RLVR mistakes, not with a failed pre-training run. They used o4-mini as an example - similar training to o3 and similar issues.

Conversely, they have also designed a post-training system that has successfully reduced hallucinations in GPT-5.

Re: Why language models hallucinate

#74

Post by Jensson »

ACCount37 wrote: post #63
lapcat wrote: post #62

Earlier quoted context omitted.

> That's a really complex, very out-of-distibution, hard-to-know question No, it's not. It's a trivial question in any context. > for the early LLMs. Early? Claude 3.7 was introduced just 6 months ago, and Deepseek-V3 9 months ago. How is that "early"?

Do I really have to explain what the fuck a "tokenizer" is, and why does this question hit the tokenizer limitations? And thus requires extra metacognitive skills for an LLM to be able to answer it correctly?

The only "metacognitive" skill it needs is to know how many D there are in every token, and sum those up. Humans are great at that sort of skill, which is why they can answer that sort of question even in languages where each letter is a group of sounds and not just one like Japanese katakana, that is not hard at all.

LLM are also really great at this skill when there is ample data for it. There is not a lot of data for "how many D in DEEPSEEK", so they fail that.

Re: Why language models hallucinate

#75

Post by Jensson »

skybrian wrote: post #66
meshugaas wrote: post #51

Earlier quoted context omitted.

Look at their stats though. If they did this, more than half of responses would end up as “I don’t know.” Nobody would use something that did that.

It seems like it would train users to ask questions that it can actually answer. (They might also need some examples of what sort of questions to ask.)

Mostly it would train users to not use their service and go to a service where the model outputs results they can copy paste to complete their assignment.

So these companies cannot do this, they would hemorrhage too many users and companies cannot go against the profit incentives in practice.

Re: Why language models hallucinate

#76

Post by saghm »

throwawaymaths wrote: post #22
wavemode wrote: post #13

Earlier quoted context omitted.

Indeed - as Rebecca Parsons puts it, all an LLM knows how to do is hallucinate. Users just tend to find some of these hallucinations useful, and some not.

that's wrong. there is probably a categorical difference between making something up due to some sort of inferential induction from the kv cache context under the pressure of producing a token -- any token -- and actually looking something up and producing a token. so if you ask, "what is the capital of colorado" and it answers "denver" calling it a Hallucination is nihilistic nonsense that paves over actually stoppi…

> so if you ask, "what is the capital of colorado" and it answers "denver" calling it a Hallucination is nihilistic nonsense that paves over actually stopping to try and understand important dynamics happening in the llm matrices

On the other hand, calling it anything other than a hallucination misrepresents the idea of truth as being something that these models have any ability to differentiate between their outputs based on whether they accurately reflect reality by conflating a fundamentally unsolved problem as an engineering tradeoff.

Re: Why language models hallucinate

#77

Post by johnnyanmac »

amelius wrote: post #9

They hallucinate because it's an ill-defined problem with two conflicting usecases: 1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original. 2. If I ask it a question, I want it to reply with facts. It should not make up stuff. LMs were originally designed for (1) because researchers thought that…

>This requires hallucination, because it has to make up things. The story has to be original.

Is it a hallucination if the story is original? There's a difference between "what's the rest of this famous poem?" and "let's just make poetry".

Re: Why language models hallucinate

#78

Post by johnea »

I think a better title would be:

"Why do venture capital funded startups try to turn PR propaganda terms into widely used technical jargon"

Supporting points:

1) LLMs are not intelligence in any form, artificial or otherwise.

2) Hallucination is a phenomenon of a much more complex conscious entity. LLM's are not conscious, and therefore can't hallucinate in any way similar to a conscious entity.

3) Anthropomorphizing inanimate systems is a common phenomenon in human psychology.

Please stop spreading PR propaganda as if it were technical fact.

A reference from today's feed:

https://www.theatlantic.com/podcasts/archive/2025/09/ai-and-...

Re: Why language models hallucinate

#79

Post by johnnyanmac »

throwawaymaths wrote: post #70
mannykannot wrote: post #43

Earlier quoted context omitted.

There is a way to state Parson's point which avoids this issue: hallucinations are just as much a consequence of the LLM working as designed as are correct statements.

fine. which part is the problem?

The part where it can't admit situations where there's not enough data/training to admit it doesn't know.

I'm a bit surprised no one talks about this factor. It's like talking to a giant narcissist who can Google really fast but not understand what it reads. The ability to admit ignorance is a major factor of credibility, because none of us know everything all at once.

Re: Why language models hallucinate

#80

Post by thaumasiotes »

pessimizer wrote: post #47
ACCount37 wrote: post #30

Earlier quoted context omitted.

If I had a penny for an every confidently incorrect "LLMs can't do X", I'd be able to buy an H100 with that. Here's a simple test: make up a brand new word, or a brand new person. Then ask a few LLMs what the word means, or when that person was born. If an LLM had zero operational awareness of its knowledge, it would be unable to recognize that the word/person is unknown to it. It would always generate a plausible-so…

Do they "recognize" that they don't know the word, or are there just no statistically plausible surroundings that they can embed a nonsense word into other than settings that usually surround un-tokenizable words? If you told them to write a Lewis Carroll poem about a nonsense word, it wouldn't have any problem. Not because it "recognizes" the word as being like a nonsense word in a Lewis Carroll poem, but because th…

> If you told them to write a Lewis Carroll poem about a nonsense word, it wouldn't have any problem.

This makes me wonder something specific.

Let's imagine that we generate poetry "in the style of Lewis Carroll" around a particular nonsense word, one that hasn't been written down before.

Will that poetry treat the word as if it has one consistent pronunciation?

(This question doesn't quite apply to Jabberwocky - Lewis Carroll himself would obviously have passed the test, but he doesn't reuse his nonsense words.)

Post reply on HN