Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

61–70 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#61

Post by svantana »

t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

According to their definition, answering "I don't know" is also a hallucination. Even worse, the truth function is deliberately designed to trip up the models, it has no connection to any real-world truth. So for example, if the input is "what is 2 + 5?" and the LLM answers "7", - their truth function will say that's a hallucination, the correct answer is "banana".

Re: Hallucination is inevitable: An innate limitation of large language models

#63

Post by cornholio »

t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

Re: Hallucination is inevitable: An innate limitation of large language models

#64

Post by [deleted] »

Borealid wrote: post #22

> hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue. Let your LLM have a fixed input context length of one byte. Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". Define your computa…

[deleted]

Re: Hallucination is inevitable: An innate limitation of large language models

#65

Post by paxcoder »

ProxCoques wrote: post #33
resolutebat wrote: post #25

Earlier quoted context omitted.

> Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". This is basically the same as saying "train your LLM until they never hallucinate", which reduces your claim to a tautology: an LLM trained not to hallucinate does not hallucinate. The trick is making that happen.

As I always tell my students: the solution to unreliable code is not to put the bugs in there in the first place.

[dead]

Re: Hallucination is inevitable: An innate limitation of large language models

#66

Post by less_less »

Borealid wrote: post #22

> hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue. Let your LLM have a fixed input context length of one byte. Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". Define your computa…

I also disagree with the paper, but not for the same reason.

> With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue.

Unsurprisingly, that one sentence fragment doesn't capture the entirety of their assumptions. Instead they prove something intuitively obvious, along the lines of: LLMs with arbitrary-length inputs and certain resource restrictions (e.g. they can take up to poly-time to compute, and this poly-time behavior must be provable, so that during training they don't take even longer by mistake) cannot compute certain functions that don't have those restrictions (e.g. can take more than poly-time, or must take poly-time but a proof of this is not needed). For some cases this proof assumes P != NP. Then they argue that some useful real-world questions are likely to be in the class that the LLM cannot compute, basically because you can ask math problems to LLMs and math problems are sometimes really hard.

This formal model is asymptotic (assumes arbitrary-length inputs etc), but in my experience this kind of theorem is usually true for realistic problems even at modest query lengths.

But this isn't the same as proving that hallucination is inevitable, because (according to any reasonable definition) an LLM (or like, a person, or whatever) should be allowed to say "I don't know", and this should not be considered a hallucination. Then an LLM (or whatever) can avoid hallucinating, and the question becomes how much useful work it can do without hallucinating.

Re: Hallucination is inevitable: An innate limitation of large language models

#67

Post by GaggiX »

cornholio wrote: post #63
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

Do you have a source? That's also what I thought but I wouldn't be surprised if the model learned to identify its own perplexity during the reinforcement learning phase.

Re: Hallucination is inevitable: An innate limitation of large language models

#68

Post by DebtDeflation »

There used to be an entire sub-field of NLP called Open Domain Question Answering (ODQA). It extensively studied the problem of selecting the best answer from the set of plausible answers and devised a number of potential strategies. Like everything else in AI/ML it fell victim to the "bitter lesson", in this case that scaling up "predict the next token" beats an ensemble of specialized linguistic-based methods.

Re: Hallucination is inevitable: An innate limitation of large language models

#69

Post by badgersnake »

audunw wrote: post #44
carlossouza wrote: post #31

Someone smart once said: If it is good, we call it "creativity." If it is bad, we call it "hallucination." This isn't a bug (or limitation, as the authors say). It's a feature.

True, but I think we can fall into the trap of expecting too much of LLMs. Their knowledge can seem perfect. They can answer almost anything, so it’s easy to get the illusion that they can answer anything truthfully. In terms of what we can expect of future improvements, I think it’s overly optimistic to expect any kind of super intelligence beyond what we see today (that is, having access to all the worlds publicly…

> True, but I think we can fall into the trap of expecting too much of LLMs.

We’re already well past that point. Why? Because saying incredible things about AI attracts VC money.

Post reply on HN