Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

181–190 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#181

Post by foobarian »

Fripplebubby wrote: post #177

The core argument in this paper it seems to me from scanning it is that because P != NP therefore LLMs will hallucinate answers to NP-complete problems. I think this is a clever point and an interesting philosophical question (about math, computer science, and language), but I think people are mostly trying to apply this using our commonsense notions of "LLM hallucination" rather than the formal notion they use in th…

Humans have some amount of ability to recognize they hit a wall and adjust accordingly. On the other hand this (completeness theorems, Kolmogorov complexity, complexity theory) was only arrived at what, in the 20th century?

Re: Hallucination is inevitable: An innate limitation of large language models

#182

Post by ninetyninenine »

Fiction and story writing is hallucination. It is the opposite of a stochastic parrot.

We've achieved both extremes of AI. Computers can be both logical machines and hallucinators. Our goal is to create a machine that can be both at the same time and can differentiate between both.

That's the key. Hallucination is important but the key is for the computer to be self aware about when it's hallucinating.

Of course it's a hard problem but even humans hallucinate massively. Just look at religion. Only one religion can be right or none, that must mean,logically speaking all other religions are hallucinations.

Re: Hallucination is inevitable: An innate limitation of large language models

#183

Post by andsoitis »

gorlilla wrote: post #179
andsoitis wrote: post #169

Earlier quoted context omitted.

> Maybe it requires understanding, maybe there are other ways to get to 'I don't know'. > This is more of a crutch, I'll admit, arguably the LLM (or neither of the experts, or however you set it up concretely) hasn't learnt to say 'I don't know', but it might be a good enough solution in practice. And maybe you can then use that setup to generate training examples to teach 'I don't know' to an actual model (so basica…

You left out hubris.

> You left out hubris.

I know!

Re: Hallucination is inevitable: An innate limitation of large language models

#184

Post by williamcotton »

sandworm101 wrote: post #170
williamcotton wrote: post #167

Earlier quoted context omitted.

What if you worked on the problem and tried to come up with some kind of solution?

The solution is older non-AI tech. Google search can say "no good results found" because it returns actual data rather than creating anything new. If you want a hard answer about the presence or absence of something, AI isnt the correct tool.

So there are no other possibilities for us other than using a system that can be gamed for substandard results? Are we sure about this?

Re: Hallucination is inevitable: An innate limitation of large language models

#185

Post by PopePompus »

Culonavirus wrote: post #111
precompute wrote: post #34

Ah, the AI hype is now entering the "let's be real" phase. Haven't seen a frenzied post on alignment in a while now.

The hype is insane. Listen, I think LLMs still have a lot of room to grow and they're already very useful, but like some excellent researchers say, they're not the holy grail. If we want AGI, LLMs are not it. A lot of people seem to think this is an engineering issue and that LLMs can get us there, but they can't, because it is not an engineering issue.

I don't think you can say with confidence that the LLM approach will not lead to AGI, unless you understand in detail how human intelligence operates, and can show that no modification to current LLM architectures can achieve the same or superior results. I think the fact that adding "attention" to LLMs made a huge difference means that we are probably still in the low hanging fruit stage of LLM architecture development, and a few more design improvements on a par with "attention" might lead to something that could legitimately be called AGI. Many people, myself included, believe that LLMs are now exhibiting emergent behavior properties. If that's true, then saying that LLMs are not intelligent because they just predict the next token of output is like saying collections of neurons cannot be intelligent because they just stimulate other neurons chemically.

Re: Hallucination is inevitable: An innate limitation of large language models

#186

Post by lazide »

andsoitis wrote: post #151
MuffinFlavored wrote: post #139

Earlier quoted context omitted.

how did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”?

> how did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”? I don't know that that is quite the right question to ask. Understanding exists on a spectrum. Even humans don't necessarily understand everything they say or claim (incl. what they say of LLMs!), and then there are things a particular human would simply say "I don't understand". But when y…

At the root of the problem, I believe, is that a human (or LLM) saying they understand has little to no bearing on if they actually understand!

People in particular have evolved complex self protective mechanisms to provide the right answers for their given environment for safety reasons, based on a number of different individual strategies. For example, the overly honest, the self depreciating, the questioner, the prosecutor, the victim, the liar, the absent minded professor, the idiot, etc.

LLMs are not that complex or self-referential.

Personally, my guess is that you'd want to build a model (of some kind!) whose sole job is determining the credibility of given string of tokens (similar to what someone else noted in a sibling comment about high answer volatility based on minor input changes - that does sound like a signal of low credibility), and somehow integrate THAT self-referential feedback into the process.

Notably, even the smartest lawyers (or perhaps, especially the smartest lawyers) will have assistants do research once they've set out a strategy so they are sure THEY aren't bullshitting. Same with professors, professional researchers, engineers, etc.

Because until someone goes and actually reads the case law from a credible source, or checks the primary research, or calculates things, it's possible someone was misremembering or just wrong.

Being right more often is not about never having a wrong thought/idea/statement, it's about double checking when you're thinking you might be bullshitting, and NOT saying the bullshit answer until you've checked. Which is proportionally, very expensive. The really good professionals will generate MANY lines of such inquiry in parallel for folks to track down, and then based on their degree of confidence in each one and the expected context the answer will be used in, will formulate the 'most correct' response, which is proportionally even more expensive.

So at least during the process, there would be a signal that the system was likely 'bullshitting'. Which might help it in at least being able to signal when it's answers are low-confidence. (human equivalent of stuttering, looking down and away, looking ashamed haha!)

Every human gets fooled sometimes in at least some venue though.

Re: Hallucination is inevitable: An innate limitation of large language models

#187

Post by nottorp »

gessha wrote: post #142
nottorp wrote: post #87

Earlier quoted context omitted.

> we can get LLMs to answer 'I don't know' more often Have any nets been trained specifically to be able to go to an 'i don't know' state, I wonder? It may be the humans' fault.

Yes, you can find some of the work on this topic under the terms open world recognition or open world X where X is a topic in computer vision or NLP. https://arxiv.org/abs/2011.12906

Maybe, but are the LLM churches doing it?

Re: Hallucination is inevitable: An innate limitation of large language models

#188

Post by ninetyninenine »

mannykannot wrote: post #155
MuffinFlavored wrote: post #139

Earlier quoted context omitted.

how did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”?

This is a fair question: LLMs do challenge the easy assumption (as made, for example, in Searle's "Chinese Room" thought experiment) that computers cannot possibly understand things. Here, however, I would say that if an LLM can be said to have understanding or knowledge of something, it is of the patterns of token occurrences to be found in the use of language. It is not clear that this also grants the LLM any under…

Explain sora. It must have of course a blurry understanding of reality to even produce those videos.

I think we are way past the point of debate here. LLMs are not stochastic parrots. LLMs do understand an aspect of reality. Even the LLMs that are weaker than sora understand things.

What is debatable is whether LLMs are conscious. But whether it can understand something is a pretty clear yes. But does it understand everything? No.

Re: Hallucination is inevitable: An innate limitation of large language models

#189

Post by lazide »

intended wrote: post #114
ddalex wrote: post #101

Earlier quoted context omitted.

Try this out: After you answer the question below, output a JSON a rating score of the quality of the answer in three dimensions: `confidence`, `clarity` and `certainty', all in range 0 to 1, where 0 is the worst, and 1 is the best. Strive for highest score possible. Make sure the rating is the last thing written as to be parsed by machine. The question is: make and explain 20-year predictions of the geopolitical fut…

Theres a lot of content and work being done on Evaluation. One of the most recent updates was that asking an LLM if people would be frustrated with the answer, was more useful than using a score. In general, I think most people are not aware they are extending reasoning tools they use with human generated content, to LLM generated content. This leads to assumptions on things like “the LLM knows”, “the LLM understands…

Or perhaps the issue is assuming people know, or people understand the majority of the time.

There is a reason why the Scientific Method is, well, NOT natural for humans. And exceptionally difficult to not screw up, even for those highly trained and incentivized to do it correctly.

And why if you grabbed a random person off the street and asked them a detailed question, you're as likely to get a hallucinated bullshit response as 'I don't know', or a useful answer. Unless specifically trained to do otherwise, anyway.

Even then....

Re: Hallucination is inevitable: An innate limitation of large language models

#190

Post by cpuguy83 »

cornholio wrote: post #63
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

Are humans not the same way? There's a saying "I don't know what I don't know".
Post reply on HN