Why language models hallucinate • Hacker News Board

Re: Why language models hallucinate

#21

Post by thaumasiotes » Sat, Sep 06, 2025, 2:28 PM UTC

roxolotl wrote: post #3

aleph_minus_one wrote: post #2

> Think about it like a multiple-choice test. If you do not know the answer but take a wild guess, you might get lucky and be right. Leaving it blank guarantees a zero. In the same way, when models are graded only on accuracy, the percentage of questions they get exactly right, they are encouraged to guess rather than say “I don’t know.” To me, this seems to be an "US-American" way of thinking about multiple-choice t…

The SAT, American college entrance examine, used to, I haven’t looked in years so maybe it still does, take away points for wrong answers and give 0 points for no answer. I’m pretty sure it was +1 for right answer, 0 for no answer, -1/4 for wrong answer.

They used to do that, but then they stopped and announced that you were better off guessing because there would be no adjustment for it.

A lot of what they do is based on public relations rather than psychometric validity.

Re: Why language models hallucinate

#22

Post by throwawaymaths » Sat, Sep 06, 2025, 2:28 PM UTC

wavemode wrote: post #13

amelius wrote: post #9

They hallucinate because it's an ill-defined problem with two conflicting usecases: 1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original. 2. If I ask it a question, I want it to reply with facts. It should not make up stuff. LMs were originally designed for (1) because researchers thought that…

Indeed - as Rebecca Parsons puts it, all an LLM knows how to do is hallucinate. Users just tend to find some of these hallucinations useful, and some not.

that's wrong. there is probably a categorical difference between making something up due to some sort of inferential induction from the kv cache context under the pressure of producing a token -- any token -- and actually looking something up and producing a token.

so if you ask, "what is the capital of colorado" and it answers "denver" calling it a Hallucination is nihilistic nonsense that paves over actually stopping to try and understand important dynamics happening in the llm matrices

Re: Why language models hallucinate

#23

Post by thaumasiotes » Sat, Sep 06, 2025, 2:29 PM UTC

ACCount37 wrote: post #8

roxolotl wrote: post #4

This seems inherently false to me. Or at least partly false. It’s reasonable to say LLMs hallucinate because they aren’t trained to say they don’t have a statistically significant answer. But there is no knowledge of correct vs incorrect in these systems. It’s all statistics so what OpenAI is describing sounds like a reasonable way to reduce hallucinations but not a way to eliminate them nor the root cause.

Is there any knowledge of "correct vs incorrect" inside you? If "no", then clearly, you can hit general intelligence without that. And if "yes", then I see no reason why an LLM can't have that knowledge crammed inside it too. Would it be perfect? Hahahaha no. But I see no reason why "good enough" could not be attained.

> And if "yes", then I see no reason why an LLM can't have that knowledge crammed inside it too.

An LLM, by definition, doesn't have such a concept. It's a model of language, hence "LLM".

Do you think the phrase just means "software"? Why?

Re: Why language models hallucinate

#24

Post by fumeux_fume » Sat, Sep 06, 2025, 2:31 PM UTC

wavemode wrote: post #13

amelius wrote: post #9

They hallucinate because it's an ill-defined problem with two conflicting usecases: 1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original. 2. If I ask it a question, I want it to reply with facts. It should not make up stuff. LMs were originally designed for (1) because researchers thought that…

Indeed - as Rebecca Parsons puts it, all an LLM knows how to do is hallucinate. Users just tend to find some of these hallucinations useful, and some not.

In the article, OpenAI defines hallucinations as "plausible but false statements generated by language models." So clearly it's not all that LLMs know how to do. I don't think Parsons is working from a useful or widely agreed upon definition of what a hallucination is which leads to these "hot takes" that just clutter and muddy up the conversation around how to reduce hallucinations to produce more useful models.

Re: Why language models hallucinate

#25

Post by throwawaymaths » Sat, Sep 06, 2025, 2:32 PM UTC

bananaflag wrote: post #15

aleph_minus_one wrote: post #2

> Think about it like a multiple-choice test. If you do not know the answer but take a wild guess, you might get lucky and be right. Leaving it blank guarantees a zero. In the same way, when models are graded only on accuracy, the percentage of questions they get exactly right, they are encouraged to guess rather than say “I don’t know.” To me, this seems to be an "US-American" way of thinking about multiple-choice t…

This is mentioned in the text: > This idea is not new. Some standardized tests have long used versions of negative marking for wrong answers or partial credit for leaving questions blank to discourage blind guessing.

there's not really an easy way to train for that at scale. a "correct" answer may not be one token, there may be multiple synonymous answers starting with different tokens, you could add five space tokens in front of the answer amd it likely shouldn't make it "wrong".

Re: Why language models hallucinate

#26

Post by ACCount37 » Sat, Sep 06, 2025, 2:33 PM UTC

wavemode wrote: post #16

ACCount37 wrote: post #8

Earlier quoted context omitted.

Is there any knowledge of "correct vs incorrect" inside you? If "no", then clearly, you can hit general intelligence without that. And if "yes", then I see no reason why an LLM can't have that knowledge crammed inside it too. Would it be perfect? Hahahaha no. But I see no reason why "good enough" could not be attained.

> Is there any knowledge of "correct vs incorrect" inside you? There is a sort of knowledge humans possess that LLMs don't (and in fact can't, without a fundamental architectural change), which is knowledge of how certain one is about something. If you ask a human a question about how something works in biology, they will be able to give you an answer as well as a sort of "epistemic" citation (i.e. the difference bet…

LLMs have that knowledge. Just not nearly enough of it. Some of it leaks through from the dataset, even in base models. The rest has to be taught on purpose.

You can get an LLM to generate a list of facts that includes hallucinations - and then give that list to another instance of the same LLM, and get it to grade how certain it is of each fact listed. The evaluation wouldn't be perfect, but it'll outperform chance.

You can make that better with the right training. Or much worse, with the wrong training. Getting an LLM to be fully aware of all the limits of its knowledge is likely to be impractical, if not outright impossible, but you can improve this awareness by a lot, and set a conservative baseline for behavior, especially in critical domains.

"Fully aware of all the limits of its knowledge" is unattainable for humans too, so LLMs are in a good company.

Re: Why language models hallucinate

#27

Post by [deleted] » Sat, Sep 06, 2025, 2:34 PM UTC

ACCount37 wrote: post #19

This mostly just restates what was already well known in the industry. Still quite useful, because, looking at the comments right now: holy shit is the "out of industry knowledge" on the topic bad! Good to have something to bring people up to speed! Good to see OpenAI's call for better performance evals - ones that penalize being confidently incorrect at least somewhat. Most current evals are "all of nothing", and th…

[deleted]

Re: Why language models hallucinate

#28

Post by fumeux_fume » Sat, Sep 06, 2025, 2:38 PM UTC

I like that OpenAI is drawing a clear line on what “hallucination” means, giving examples, and showing practical steps for addressing them. The post isn’t groundbreaking, but it helps set the tone for how we talk about hallucinations.

What bothers me about the hot takes is the claim that “all models do is hallucinate.” That collapses the distinction entirely. Yes, models are just predicting the next token—but that doesn’t mean all outputs are hallucinations. If that were true, it’d be pointless to even have the term, and it would ignore the fact that some models hallucinate much less than others because of scale, training, and fine-tuning.

That’s why a careful definition matters: not every generation is a hallucination, and having good definitions let us talk about the real differences.

Re: Why language models hallucinate

#29

Post by hodgehog11 » Sat, Sep 06, 2025, 2:43 PM UTC

amelius wrote: post #9

They hallucinate because it's an ill-defined problem with two conflicting usecases: 1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original. 2. If I ask it a question, I want it to reply with facts. It should not make up stuff. LMs were originally designed for (1) because researchers thought that…

I don't agree that it is an ill-defined problem, since we can design separate models to excel in each of these two tasks. For a "factual" LLM, if the output is a verifiable statement, it should be correct. Otherwise it "hallucinates". But since an LLM can't know everything, a better approach is to effectively state its own uncertainty so that it avoids making definitive statements with low confidence.

Re: Why language models hallucinate

#30

Post by ACCount37 » Sat, Sep 06, 2025, 2:46 PM UTC

thaumasiotes wrote: post #23

ACCount37 wrote: post #8

Earlier quoted context omitted.

Is there any knowledge of "correct vs incorrect" inside you? If "no", then clearly, you can hit general intelligence without that. And if "yes", then I see no reason why an LLM can't have that knowledge crammed inside it too. Would it be perfect? Hahahaha no. But I see no reason why "good enough" could not be attained.

> And if "yes", then I see no reason why an LLM can't have that knowledge crammed inside it too. An LLM, by definition, doesn't have such a concept. It's a model of language, hence "LLM". Do you think the phrase just means "software"? Why?

If I had a penny for an every confidently incorrect "LLMs can't do X", I'd be able to buy an H100 with that.

Here's a simple test: make up a brand new word, or a brand new person. Then ask a few LLMs what the word means, or when that person was born.

If an LLM had zero operational awareness of its knowledge, it would be unable to recognize that the word/person is unknown to it. It would always generate a plausible-sounding explanation for what the word might mean, the same exact way it does for the word "carrot". Or a plausible-sounding birth date, the way it does for the person "Abraham Lincoln".

In practice, most production grade LLMs would recognize that a word or a person is unknown to them.

This is a very limited and basic version of the desirable "awareness of its own knowledge" - and one that's already present in current LLMs! Clearly, there's room for improved self-awareness.