Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why language models hallucinate

openai.com

Post reply on HN
Print view View original

181–190 of 242 posts

Re: Why language models hallucinate

#181

Post by esafak »

vrighter wrote: post #177
fumeux_fume wrote: post #28

I like that OpenAI is drawing a clear line on what “hallucination” means, giving examples, and showing practical steps for addressing them. The post isn’t groundbreaking, but it helps set the tone for how we talk about hallucinations. What bothers me about the hot takes is the claim that “all models do is hallucinate.” That collapses the distinction entirely. Yes, models are just predicting the next token—but that do…

if you insist that they are different, then please find one logical, non-subjective, way to distinguish between a hallucination and not-a-hallucination. Looking at the output and deciding "this is clearly wrong" does not count. No vibes.

> Looking at the output and deciding "this is clearly wrong" does not count.

You need the ground truth to be able to make that determination, so using your knowledge does count. If you press the model to answer even when it does not know, you get confabulation. What today's models lack is the ability to measure their confidence, so they know when to abstain.

Re: Why language models hallucinate

#182

Post by beeflet »

crystal_revenge wrote: post #150
rhubarbtree wrote: post #105

I find this rather oddly phrased. LLMs hallucinate because they are language models. They are stochastic models of language. They model language, not truth. If the “truthy” responses are common in their training set for a given prompt, you might be more likely to get something useful as output. Feels like we fell into that idea and said - ok this is useful as an information retrieval tool. And now we use RL to reinfo…

People also tend not to understand the absurdity of assuming that we can make LLMs stop hallucinating. It would imply not only that truth is absolutely objective , but that it exists on some smooth manifold which language can be mapped to. That means there would be some high dimensional surface representing "all true things". Any fact could be trivially resolved as "true" or "false" simply by exploring whether or not…

Maybe if a language model was so absolutely massive, it could enough to simulate the entire universe and determine your social security number

Re: Why language models hallucinate

#183

Post by apsurd »

jdietrich wrote: post #169
crystal_revenge wrote: post #150

Earlier quoted context omitted.

People also tend not to understand the absurdity of assuming that we can make LLMs stop hallucinating. It would imply not only that truth is absolutely objective , but that it exists on some smooth manifold which language can be mapped to. That means there would be some high dimensional surface representing "all true things". Any fact could be trivially resolved as "true" or "false" simply by exploring whether or not…

>People also tend not to understand the absurdity of assuming that we can make LLMs stop hallucinating. It would imply not only that truth is absolutely objective, but that it exists on some smooth manifold which language can be mapped to. Frankly, this is a silly line of argument. There is a vast spectrum between regularly inventing non-existent citations and total omniscience. "We can't define objective truth" isn'…

You're not being charitable with the take. seems like you just switched "objective truth" with your flavor: "error rate"

what is an error? how does the llm "know"?

wikipedia example is good. i'd say its "truth" is based on human curated consensus. everyone gets that. what i don't get what's the llm analog? as you state, it's just about making the error rate go down, ok so what is an error? does it require human in the loop?

Re: Why language models hallucinate

#184

Post by skydhash »

jdietrich wrote: post #169
crystal_revenge wrote: post #150

Earlier quoted context omitted.

People also tend not to understand the absurdity of assuming that we can make LLMs stop hallucinating. It would imply not only that truth is absolutely objective , but that it exists on some smooth manifold which language can be mapped to. That means there would be some high dimensional surface representing "all true things". Any fact could be trivially resolved as "true" or "false" simply by exploring whether or not…

>People also tend not to understand the absurdity of assuming that we can make LLMs stop hallucinating. It would imply not only that truth is absolutely objective, but that it exists on some smooth manifold which language can be mapped to. Frankly, this is a silly line of argument. There is a vast spectrum between regularly inventing non-existent citations and total omniscience. "We can't define objective truth" isn'…

The thing is, for a lot of tasks, a formal method (either algorithmic or simulation) can be very efficient to create and run with more reliable results. And for a lot of cases, creating a simpler and smaller model with other ML techniques can be as good or better than LLMs.

There's still no justification for the whole investment craze in LLMs.

Re: Why language models hallucinate

#185

Post by red75prime »

gary_0 wrote: post #153
mqus wrote: post #151

Earlier quoted context omitted.

Well, no. The article pretty much says that any arbitrary statement can be mapped to {true, false, I don't know}. This is still not 100% accurate, but at least something that seems reachable. The model should just be able to tell unknowns, not be able to verify every single fact.

Determining a statement's truth (or if it's outside the system's knowledge) is an old problem in machine intelligence, with whole subfields like knowledge graphs and such, and it's NOT a problem LLMs were originally meant to address at all. LLMs are text generators that are very good at writing a book report based on a prompt and the patterns learned from the training corpus, but it's an entirely separate problem to…

> And that problem is one that the AI field has already spent 60 years on

I hope you don't think that the solutions will be a closed-form expression. The solution should involve exploration and learning. The things that LLMs are instrumental in, you know.

Re: Why language models hallucinate

#186

Post by riwsky »

beeflet wrote: post #182
crystal_revenge wrote: post #150

Earlier quoted context omitted.

People also tend not to understand the absurdity of assuming that we can make LLMs stop hallucinating. It would imply not only that truth is absolutely objective , but that it exists on some smooth manifold which language can be mapped to. That means there would be some high dimensional surface representing "all true things". Any fact could be trivially resolved as "true" or "false" simply by exploring whether or not…

Maybe if a language model was so absolutely massive, it could enough to simulate the entire universe and determine your social security number

42

Re: Why language models hallucinate

#187

Post by riwsky »

zbentley wrote: post #145
ninetyninenine wrote: post #121

Earlier quoted context omitted.

>But an LLM is not answering "what is truth?". It's "answering" "what does an answer to the question "what is truth?" look like?". You don't actually know this right? You said what I'm saying is theoretically possible so you're contradicting what you're saying. >You're anthropomorphizing in using terms like "lying to us" or "know the truth". Yes, it's theoretically possible I suppose that they've secretly obtained so…

I think your assessment of the academic take on AI is wrong. We have a rather thorough understanding of the how/why of the mechanisms of LLMs, even if after training their results sometimes surprise us. Additionally, there is a very large body of academic research that digs into how LLMs seem to understand concepts and truths and, sure enough, examples of us making point edits to models to change the “facts” that the…

Here’s where you're clearly wrong. The correct favorite in that corpus is Golden Gate Claude: https://www.anthropic.com/news/golden-gate-claude

Re: Why language models hallucinate

#188

Post by renewiltord »

robotcapital wrote: post #170

It’s interesting that most of the comments here read like projections of folk-psych intuitions. LLMs hallucinate because they “think” wrong, or lack self-awareness, or should just refuse. But none of that reflects how these systems actually work. This is a paper from a team working at the state of the art, trying to explain one of the biggest open challenges in LLMs, and instead of engaging with the mechanisms and ev…

It's always the most low-brow takes as well. But the majority of Hacker News commentators "hallucinate" most of their comments in the first place, since they simply regurgitate the top answers based on broad bucketing of subject matter.

Facebook? "Steal your data"

Google? "Kill your favourite feature"

Apple? "App Store is enemy of the people"

OpenAI? "More like ClosedAI amirite"

Re: Why language models hallucinate

#189

Post by gary_0 »

red75prime wrote: post #185
gary_0 wrote: post #153

Earlier quoted context omitted.

Determining a statement's truth (or if it's outside the system's knowledge) is an old problem in machine intelligence, with whole subfields like knowledge graphs and such, and it's NOT a problem LLMs were originally meant to address at all. LLMs are text generators that are very good at writing a book report based on a prompt and the patterns learned from the training corpus, but it's an entirely separate problem to…

> And that problem is one that the AI field has already spent 60 years on I hope you don't think that the solutions will be a closed-form expression. The solution should involve exploration and learning. The things that LLMs are instrumental in, you know.

Of course not, expert systems were abandoned decades ago for good reason. But LLMs are only one kind of ANN. Unfortunately, when all you have is a hammer...

Re: Why language models hallucinate

#190

Post by simianparrot »

thisoneisreal wrote: post #161
rhubarbtree wrote: post #105

I find this rather oddly phrased. LLMs hallucinate because they are language models. They are stochastic models of language. They model language, not truth. If the “truthy” responses are common in their training set for a given prompt, you might be more likely to get something useful as output. Feels like we fell into that idea and said - ok this is useful as an information retrieval tool. And now we use RL to reinfo…

This strikes me as a perfect description of the core problem. Whenever I think about this, what sticks out to me is that other animals do all sorts of things that look like "intelligence," or at least cognition, and they do it totally without language. My cat clearly recognizes objects, assigns them different values ("scary," "tasty," "fun to play with"), interacts with them in some kind of loop, even predicts their…

I’ve always thought everyone agreed language was a lossy but useful method of compression for sharing inner concepts and ideas. That my conscious thoughts are “in a language” doesn’t mean my reasoning and entire being interacts with the world using language.

I’m only “thinking in language” when I’m practicing compressing my intent into a shareable format. I don’t think about the majority of highly complex interactions I have with the physical world throughout the day.

As a child did you need to be able to explain in language how the physics of a swing works to be able to use it? Did other kids have to explain it to you in detailed language for you to pick up on how to move your body to do complex tasks?

No. In fact exactly because our compression and decompression of language is even more limited as children, we rely more heavily on raw observation and mimicry of actions occurring in reality itself.

The very idea that a language model can recreate everything we do from the lossy and compressed languages we use to share limited descriptions of much more complex intentions and actions is fundamentally flawed and oversimplified.

Post reply on HN