Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why language models hallucinate

openai.com

Post reply on HN
Print view View original

221–230 of 242 posts

Re: Why language models hallucinate

#221

Post by player1234 »

ComplexSystems wrote: post #107
saghm wrote: post #76

Earlier quoted context omitted.

> so if you ask, "what is the capital of colorado" and it answers "denver" calling it a Hallucination is nihilistic nonsense that paves over actually stopping to try and understand important dynamics happening in the llm matrices On the other hand, calling it anything other than a hallucination misrepresents the idea of truth as being something that these models have any ability to differentiate between their outputs…

It isn't a hallucination because that isn't how the term is defined. The term "hallucination" refers, very specifically, to "plausible but false statements generated by language models." At the end of the day, the goal is to train models that are able to differentiate between true and false statements, at least to a much better degree than they can now, and the linked article seems to have some very interesting sugge…

Why use a word that you have to redefine the meaning of? The answer is to deceive.

Re: Why language models hallucinate

#222

Post by player1234 »

ninetyninenine wrote: post #39
amelius wrote: post #9

They hallucinate because it's an ill-defined problem with two conflicting usecases: 1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original. 2. If I ask it a question, I want it to reply with facts. It should not make up stuff. LMs were originally designed for (1) because researchers thought that…

Did you read the article? You’re going on some generic tangent and regurgitating the same spiel about LLMs that you see all over the internet. I mean it’s plain that you have an orthogonal (though generic) opinion on why LLMs hallucinate but how does that relate to the article? How does your opinion which you blatantly just dropped as if it’s the final opinion override the opinion of the article? Seems off topic hone…

[flagged]

Re: Why language models hallucinate

#223

Post by swores »

druskacik wrote: post #209
fumeux_fume wrote: post #28

I like that OpenAI is drawing a clear line on what “hallucination” means, giving examples, and showing practical steps for addressing them. The post isn’t groundbreaking, but it helps set the tone for how we talk about hallucinations. What bothers me about the hot takes is the claim that “all models do is hallucinate.” That collapses the distinction entirely. Yes, models are just predicting the next token—but that do…

I like this quote: 'Everything an LLM outputs is a hallucination. It's just that some of those hallucinations are true.'

To me that seems as pointless as saying "everything a person sees is a hallucination, it's just some of those hallucinations are true". Sure, technically whenever we see anything it's actually our brain interpreting how light bounces off stuff and combining that with the mental models we have of the world to produce an image in our mind of what we're looking at... but if we start calling everything we see a hallucination, there's no longer any purpose in having that word.

So instead of being that pedantic, we decided that "hallucination" only applies to when what our brain thinks we see does not match reality, so now hallucination is actually a useful word to use. Equally with LLMs, when people talk about hallucinations part of the definition includes that the output be incorrect in some way. If you just go with your quote's way of thinking about it, then once again the word loses all purpose and we can just scrap it since it now means exactly the same thing as "all LLM output".

Re: Why language models hallucinate

#224

Post by awongh »

From an outsider perspective these kinds of insights make me think: Is it just a coincidence that a lot of the recent innovations in the space look very common sense in hindsight:

- if we train the model to "think" through the answer, we get better results - if we train the model to say "I don't know" when it's not sure we get less hallucinations

Is it just confirmation bias or do these common sense approaches work in on LLMs in other ways?

Re: Why language models hallucinate

#225

Post by a2128 »

kingstnap wrote: post #53

There is this deeply wrong part of this paper that no one has mentioned: The model head doesn't hallucinate. The sampler does. If you ask an LLM when x was born and it doesn't know. And you take a look at the actual model outputs which is a probability distribution over tokens. IDK is cleanly represented as a uniform probability Jan 1 to Dec 31 If you ask it to answer a multiple choice question and it doesn't know. I…

This is only true if you have a pretrained base model trained on infinite true data with no bias. In practice it will have picked up some bias, maybe it encountered more famous "James" birthdays in January and on a digit starting with 2, so Jan 2 and Jan 20-29 has a higher probability than all. But finetuning and especially RL completely breaks these probabilities as a measure of certainty because the goals shift from generally modelling text to something else entirely.

Re: Why language models hallucinate

#226

Post by 1718627440 »

swores wrote: post #223
druskacik wrote: post #209

Earlier quoted context omitted.

I like this quote: 'Everything an LLM outputs is a hallucination. It's just that some of those hallucinations are true.'

To me that seems as pointless as saying "everything a person sees is a hallucination, it's just some of those hallucinations are true". Sure, technically whenever we see anything it's actually our brain interpreting how light bounces off stuff and combining that with the mental models we have of the world to produce an image in our mind of what we're looking at... but if we start calling everything we see a hallucina…

> everything a person sees is a hallucination, it's just some of those hallucinations are true

Except it's not. People can have hallucinations that are true (dreams), but most perception isn't generated by your brain, but comes from the outside.

Re: Why language models hallucinate

#227

Post by 1718627440 »

esafak wrote: post #181
vrighter wrote: post #177

Earlier quoted context omitted.

if you insist that they are different, then please find one logical, non-subjective, way to distinguish between a hallucination and not-a-hallucination. Looking at the output and deciding "this is clearly wrong" does not count. No vibes.

> Looking at the output and deciding "this is clearly wrong" does not count. You need the ground truth to be able to make that determination, so using your knowledge does count. If you press the model to answer even when it does not know, you get confabulation. What today's models lack is the ability to measure their confidence, so they know when to abstain.

And having no ground truth is what defines a hallucination.

Re: Why language models hallucinate

#228

Post by player1234 »

robotcapital wrote: post #174
zahlman wrote: post #172

Earlier quoted context omitted.

Calling it a "hallucination" is anthropomorphizing too much in the first place, so....

Right, that’s kind of my point. We call it “hallucination” because we don’t understand it, but need a shorthand to convey the concept. Here’s a paper trying to demystify it so maybe we don’t need to make up anthropomorphized theories.

[dead]

Re: Why language models hallucinate

#229

Post by player1234 »

renewiltord wrote: post #188
robotcapital wrote: post #170

It’s interesting that most of the comments here read like projections of folk-psych intuitions. LLMs hallucinate because they “think” wrong, or lack self-awareness, or should just refuse. But none of that reflects how these systems actually work. This is a paper from a team working at the state of the art, trying to explain one of the biggest open challenges in LLMs, and instead of engaging with the mechanisms and ev…

It's always the most low-brow takes as well. But the majority of Hacker News commentators "hallucinate" most of their comments in the first place, since they simply regurgitate the top answers based on broad bucketing of subject matter. Facebook? "Steal your data" Google? "Kill your favourite feature" Apple? "App Store is enemy of the people" OpenAI? "More like ClosedAI amirite"

[flagged]

Re: Why language models hallucinate

#230

Post by player1234 »

ninetyninenine wrote: post #178
kdnvk wrote: post #104

Earlier quoted context omitted.

Did you read the linked paper?

The majority of people on this thread didn't even click on the link. People are so taken by their own metaphysical speculations of what an LLM is. Like literally the inventor of the LLM wrote an article and everyone is criticizing that article without even reading it. Most of these people have never built an LLM before either.

[dead]
Post reply on HN