Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why language models hallucinate

openai.com

Post reply on HN
Print view View original

81–90 of 242 posts

Re: Why language models hallucinate

#81

Post by RugnirViking »

throwawaymaths wrote: post #69
ACCount37 wrote: post #42

Earlier quoted context omitted.

Yes, it's not nearly as easy as "just fix the evals". But better evals are still helpful, because they reward LLM vendors for trying to do the very-hard-to-do thing. Instead of rewarding them for training an LLM that's really good at emitting 7% confidence guesses.

you're missing the point. SAT multiple choice negatives for random guesses, fine, you could trivially use this sort of a strategy for assigning cost functions to a classifier and backpropagate. how do you give negative weight to a wrong answer when training a transformer?

isn't this just related to the question "how do you train a transformer"? you give it wrong examples, and use optimization algorithms to move away from that kind of completions

Re: Why language models hallucinate

#82

Post by wnoise »

ACCount37 wrote: post #67
wavemode wrote: post #58

Earlier quoted context omitted.

No, LLMs don't have that knowledge. They can't inspect their own weights and examine the contents. It's a fundamental limitation of the technology. The sort of training you're talking about is content like, "ChatGPT was trained on research papers in the area of biology. It possesses knowledge of A, B, and C. It does not possess knowledge of X, Y and Z." But this merely creates the same problem in a loop - given a que…

Humans can't "inspect their own weights and examine the contents" either. No human has ever managed to read out his connectome without external instrumentation. There were entire human civilizations that thought that the seat of consciousness was the heart - which, for creatures that claim to know how their own minds work, is a baffling error to make. LLMs are quite similar in that to humans. They, too, have no idea…

No, humans can't inspect their own weights either -- but we're not LLMs and don't store all knowledge implicitly as probabilities to output next token. It's pretty clear that we also store some knowledge explicitly, and can include context of that knowledge.

(To be sure, there are plenty of cases where it is clear that we are only making up stories after the fact about why we said or did something. But sometimes we do actually know and that reconstruction is accurate.)

Re: Why language models hallucinate

#83

Post by [deleted] »

johnnyanmac wrote: post #77
amelius wrote: post #9

They hallucinate because it's an ill-defined problem with two conflicting usecases: 1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original. 2. If I ask it a question, I want it to reply with facts. It should not make up stuff. LMs were originally designed for (1) because researchers thought that…

>This requires hallucination, because it has to make up things. The story has to be original. Is it a hallucination if the story is original? There's a difference between "what's the rest of this famous poem?" and "let's just make poetry".

[deleted]

Re: Why language models hallucinate

#84

Post by johnnyanmac »

mpweiher wrote: post #32
fumeux_fume wrote: post #24

Earlier quoted context omitted.

In the article, OpenAI defines hallucinations as "plausible but false statements generated by language models." So clearly it's not all that LLMs know how to do. I don't think Parsons is working from a useful or widely agreed upon definition of what a hallucination is which leads to these "hot takes" that just clutter and muddy up the conversation around how to reduce hallucinations to produce more useful models.

They just redefined the term so that they no longer call hallucinations that are useful hallucinations. But the people who say everything LLMs do is hallucinate clearly also make that distinction, they just refuse to rename the useful hallucinations. "How many legs does a dog have if you call his tail a leg? Four. Saying that a tail is a leg doesn't make it a leg." -- Abraham Lincoln

I'd say a humans ability to reason with theoretical situations like this is our very core of creativity and intelligence, though. This quote makes sense for a policy maker, but not a scientist.

Now granted, we also need to back up those notions with rigorous testing and observation, but those "if a tail is a leg" theoretical is the basis of the reasoning.

Re: Why language models hallucinate

#85

Post by kouru225 »

AI hallucination is an inherent problem of AI. You can mitigate it, but the whole point of AI IS hallucination. If the result is useful to us, we don’t call it anything. If the result is not useful to us, we call it “hallucination”

Re: Why language models hallucinate

#86

Post by cjauvin »

amelius wrote: post #9

They hallucinate because it's an ill-defined problem with two conflicting usecases: 1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original. 2. If I ask it a question, I want it to reply with facts. It should not make up stuff. LMs were originally designed for (1) because researchers thought that…

If you consider this from the angle of Wittgenstein's "language games", you could say that the problem would be "simply" to distinguish between these two, quite different, language games, and act accordingly.

Re: Why language models hallucinate

#87

Post by catlifeonmars »

ninetyninenine wrote: post #44
saghm wrote: post #37

Earlier quoted context omitted.

This is a a super helpful way of putting it. I've tried to explain to my less technical friends and relatives that from the standpoint of an LLM, there's no concept of "truth", and that all it basically just comes up with the shape of what a response should look like and then fills in the blanks with pretty much anything it wants. My success in getting the point across has been mixed, so I'll need to try out this muc…

But this explanation doesn’t fully characterize it does it? Have the LLM talk about what “truth” is and the nature of LLM hallucinations and it can cook up an explanation that demonstrates it completely understands the concepts. Additionally when the LLM responds MOST of the answers are true even though quite a bit are wrong. If it had no conceptual understanding of truth than the majority of its answers would be wro…

> Have the LLM talk about what “truth” is and the nature of LLM hallucinations and it can cook up an explanation that demonstrates it completely understands the concepts.

There is not necessarily a connection between what an LLM understands and what it says. It’s totally possible to emit text that is logically consistent without understanding. As a trivial example, just quote from a physics textbook.

I’m not saying your premise is necessarily wrong: that LLMs can understand the difference between truth and falsehood. All I’m saying is you can’t infer that from the simple test of talking to an LLM.

Re: Why language models hallucinate

#89

Post by catlifeonmars »

fumeux_fume wrote: post #28

I like that OpenAI is drawing a clear line on what “hallucination” means, giving examples, and showing practical steps for addressing them. The post isn’t groundbreaking, but it helps set the tone for how we talk about hallucinations. What bothers me about the hot takes is the claim that “all models do is hallucinate.” That collapses the distinction entirely. Yes, models are just predicting the next token—but that do…

So there are two angles to this:

- From the perspective of LLM research/engineering, saying all LLM generation is hallucination is not particularly useful. It’s meaningless for the problem space.

- From the perspective of AI research/engineering in general (not LLM specific) it can be useful to consider architectures that do not rely on hallucination in the second sense.

Re: Why language models hallucinate

#90

Post by furyofantares »

amelius wrote: post #9

They hallucinate because it's an ill-defined problem with two conflicting usecases: 1. If I tell it the first two lines of a story, I want the LLM to complete the story. This requires hallucination, because it has to make up things. The story has to be original. 2. If I ask it a question, I want it to reply with facts. It should not make up stuff. LMs were originally designed for (1) because researchers thought that…

Wanting it to pick between those modes based on what you asked for is not remotely ill-defined.

But even if we restricted ourselves to the case of factual queries, the article discusses why training in a certain way would still produce hallucinations, and how to change the training method to reduce this.

Like many of the other responses here, your dismissal doesn't really address any of the content of the article, just the title.

Post reply on HN