Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why language models hallucinate

openai.com

Post reply on HN
Print view View original

151–160 of 242 posts

Re: Why language models hallucinate

#151

Post by mqus »

crystal_revenge wrote: post #150
rhubarbtree wrote: post #105

I find this rather oddly phrased. LLMs hallucinate because they are language models. They are stochastic models of language. They model language, not truth. If the “truthy” responses are common in their training set for a given prompt, you might be more likely to get something useful as output. Feels like we fell into that idea and said - ok this is useful as an information retrieval tool. And now we use RL to reinfo…

People also tend not to understand the absurdity of assuming that we can make LLMs stop hallucinating. It would imply not only that truth is absolutely objective , but that it exists on some smooth manifold which language can be mapped to. That means there would be some high dimensional surface representing "all true things". Any fact could be trivially resolved as "true" or "false" simply by exploring whether or not…

Well, no. The article pretty much says that any arbitrary statement can be mapped to {true, false, I don't know}. This is still not 100% accurate, but at least something that seems reachable. The model should just be able to tell unknowns, not be able to verify every single fact.

Re: Why language models hallucinate

#152

Post by mqus »

I think one of the main problems is the dataset it is trained on, which is written text. How much answers with statements are in a given text, compared to a "I don't know"? I think the "I don't know"s are much less represented. Now go anywhere on the internet where someone asks a question (the typical kind of content LLMs are trained on) and the problem is even bigger. You either get no textual answer or someone that gives some answer (that might even be false). You never get an answer like "I don't know", especially for questions that are shouted into the void (compared to asking a certain person). And it makes sense. I wouldn't start to answer every stackoverflow question with "I don't know" tomorrow, it would just be spam.

For me, as a layman (with no experience at all about how this actually works), this seems to be the cause. Can we work around this? Maybe.

Re: Why language models hallucinate

#153

Post by gary_0 »

mqus wrote: post #151
crystal_revenge wrote: post #150

Earlier quoted context omitted.

People also tend not to understand the absurdity of assuming that we can make LLMs stop hallucinating. It would imply not only that truth is absolutely objective , but that it exists on some smooth manifold which language can be mapped to. That means there would be some high dimensional surface representing "all true things". Any fact could be trivially resolved as "true" or "false" simply by exploring whether or not…

Well, no. The article pretty much says that any arbitrary statement can be mapped to {true, false, I don't know}. This is still not 100% accurate, but at least something that seems reachable. The model should just be able to tell unknowns, not be able to verify every single fact.

Determining a statement's truth (or if it's outside the system's knowledge) is an old problem in machine intelligence, with whole subfields like knowledge graphs and such, and it's NOT a problem LLMs were originally meant to address at all.

LLMs are text generators that are very good at writing a book report based on a prompt and the patterns learned from the training corpus, but it's an entirely separate problem to go through that book report statement by statement and determine if each one is true/false/unknown. And that problem is one that the AI field has already spent 60 years on, so there's a lot of hubris in assuming you can just solve that and bolt it onto the side of GPT-5 by next quarter.

Re: Why language models hallucinate

#154

Post by ekianjo »

pawelmurias wrote: post #132
utyop22 wrote: post #116

Earlier quoted context omitted.

The reality is, language itself does not capture the entirety of what is really going on. And I'd get argue its the poorest way of expressing - but one that enables transmission through various mediums efficiently on a cost basis. E.g. when I explain a concept, what comes to my mind is not a string of letters and words. There is a mix of imagery and even sounds that I may have acquired from learning about a concept -…

I would assume most people use native subtitles when it's hard to understand what words the actors said.

Yeah because modern filmmakers make it very hard to hear dialogs for some reason and actors are encouraged to mumble. If I remember correctly even Nolan admitted it.

Re: Why language models hallucinate

#155

Post by raincole »

ninetyninenine wrote: post #39
amelius wrote: post #9

They hallucinate because it's an ill-defined problem with two conflicting usecases: 1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original. 2. If I ask it a question, I want it to reply with facts. It should not make up stuff. LMs were originally designed for (1) because researchers thought that…

Did you read the article? You’re going on some generic tangent and regurgitating the same spiel about LLMs that you see all over the internet. I mean it’s plain that you have an orthogonal (though generic) opinion on why LLMs hallucinate but how does that relate to the article? How does your opinion which you blatantly just dropped as if it’s the final opinion override the opinion of the article? Seems off topic hone…

Generally HN commenters don't read the article. They use the title as a prompt to express their opinions on a specific topic.

Re: Why language models hallucinate

#157

Post by mountainriver »

Jensson wrote: post #68
mountainriver wrote: post #40

Earlier quoted context omitted.

There is knowledge of correct and incorrect, that’s what loss is, there are just often many possible answers to a question. This is the same reason that RLVR works. There is just right one answer and LLMs learn this fairly well but not perfectly (yet)

> There is knowledge of correct and incorrect, that’s what loss is Loss is only correctness in terms of correct language, not correct knowledge. It correlates with correct knowledge, but that is all, that correlation is why LLM is useful for tasks at all but we still don't have a direct measure for correct knowledge in the models. So for language tasks loss is correctness, so for things like translations LLM are extr…

We do with RLVR and that works, there is only one answer, it has to find it. LLMs are often also trained on factual information, and tested on that.

If the knowledge can be represented in text then they can learn it, if it can't then we need a multimodal model.

Re: Why language models hallucinate

#158

Post by numeri »

kingstnap wrote: post #53

There is this deeply wrong part of this paper that no one has mentioned: The model head doesn't hallucinate. The sampler does. If you ask an LLM when x was born and it doesn't know. And you take a look at the actual model outputs which is a probability distribution over tokens. IDK is cleanly represented as a uniform probability Jan 1 to Dec 31 If you ask it to answer a multiple choice question and it doesn't know. I…

This isn't right – calibration (informally, the degree to which certainty in the model's logits correlates with its chance of getting an answer correct) is well studied in LLMs of all sizes. LLMs are not (generally) well calibrated.

Re: Why language models hallucinate

#159

Post by asats »

rhubarbtree wrote: post #105

I find this rather oddly phrased. LLMs hallucinate because they are language models. They are stochastic models of language. They model language, not truth. If the “truthy” responses are common in their training set for a given prompt, you might be more likely to get something useful as output. Feels like we fell into that idea and said - ok this is useful as an information retrieval tool. And now we use RL to reinfo…

Exactly. I always found it strange when people assume that "hallucinations" are just some sort of a bug in the system, as if by you tweaking some code or training modality will produce an oracle of absolute truth incapable of making mistakes.
Post reply on HN