Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

151–160 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#151

Post by andsoitis »

MuffinFlavored wrote: post #139
andsoitis wrote: post #132

Earlier quoted context omitted.

> the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). To answer "I don't know" requires one to know when you know. To know when you know in turn requires understanding.

how did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”?

> how did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”?

I don't know that that is quite the right question to ask.

Understanding exists on a spectrum. Even humans don't necessarily understand everything they say or claim (incl. what they say of LLMs!), and then there are things a particular human would simply say "I don't understand".

But when you ask a human "can you understand things?" you will get an unequivocal Yes!

Ask that same question of an LLM and what does it say? I don't think any of them currently respond with a simple or even qualified "Yes". Now, some might claim that one day an LLM will cross that threshold and say "Yes!" but we can safely leave that off to the side for a future debate if it ever happens.

General note: it is worth separating out things like "understanding", "knowledge", "intelligence", "common sense", "wisdom", "critical thinking", etc. While they might all be related in some ways and even overlap, it does not follow that if you show high performance in one that you automatically excel in each of the other. I know many people who anyone would say are highly intelligent but lack common sense, etc.

Re: Hallucination is inevitable: An innate limitation of large language models

#152

Post by Certhas »

phh wrote: post #130
intended wrote: post #115

Earlier quoted context omitted.

The fact that a human chooses not to do remember their citations, does not mean they lack the ability. This argument comes up many times “people don’t do this” - but that is a question of frequency, not whether or not people are capable.

LLMs are capable as well if you give them access to the internet though

Humans did research and remembered sources before the Internet was a thing.

But also, can you give an example where an LLM with access to the Internet can find a primary source?

I don't think learning to refer to sources is something inherently impossible for LLMs, but it is very different to the kind of implicit knowledge they seem to excel at.

Re: Hallucination is inevitable: An innate limitation of large language models

#153

Post by Cacti »

macrolime wrote: post #85

They define an LLM as "a probabilistic model of a string that conditions the output at time t based on all the tokens that come before it in the string". I think that definition is wide enough to include human intelligence, so their finding should be equally valid for humans.

When we can define and measure intelligence, perhaps these discussions will be meaningful.

Re: Hallucination is inevitable: An innate limitation of large language models

#154

Post by moooo99 »

magicalhippo wrote: post #122
moooo99 wrote: post #96

Earlier quoted context omitted.

> I think that definition is wide enough to include human intelligence, so their finding should be equally valid for humans. Which is definitely true. Human memory and the ability to correctly recall things we though we remembered is affected by a whole bunch of things and at times very unreliable. However, human intelligence, unlike LLMs, is not limited to recalling information we once learned. We are also able to d…

> We are also able to do logical reasoning This is effectively like coming up with an algorithm and then executing it. So how good/bad are these LLMs if you asked them to generate say a LUA script to compute the answer, ala counting occurrences problem mentioned in a different comment, and then pass that off to a LUA interpreter to get the answer?

> counting occurrences problem mentioned in a different comment, and then pass that off to a LUA interpreter to get the answer?

I think this is a sensible approach in some problem domains with software development being a particularly good example. But I think this approach quickly falls apart as soon as your „definitely right answer“ involves real world interaction.

And if one thinks about it, most of the value any company derives comes down to some sort of real world interaction, wether directly or by proxy.

Re: Hallucination is inevitable: An innate limitation of large language models

#155

Post by mannykannot »

MuffinFlavored wrote: post #139
andsoitis wrote: post #132

Earlier quoted context omitted.

> the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). To answer "I don't know" requires one to know when you know. To know when you know in turn requires understanding.

how did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”?

This is a fair question: LLMs do challenge the easy assumption (as made, for example, in Searle's "Chinese Room" thought experiment) that computers cannot possibly understand things. Here, however, I would say that if an LLM can be said to have understanding or knowledge of something, it is of the patterns of token occurrences to be found in the use of language. It is not clear that this also grants the LLM any understanding that this language refers to an external world which operates in response to causes which are independent of what is or might be said about it.

Re: Hallucination is inevitable: An innate limitation of large language models

#156

Post by rdedev »

moyix wrote: post #123
cornholio wrote: post #63

Earlier quoted context omitted.

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

Seems to be contradicted by this paper, no? https://arxiv.org/abs/2207.05221

I don't think the paper addresses the question of self reflection. Like it can reflect on the question and answer pairs in its prompt but it didn't know that it created them in the first place or use that information to update it's priors, things humans can do

Re: Hallucination is inevitable: An innate limitation of large language models

#157

Post by Borealid »

mepiethree wrote: post #81
Borealid wrote: post #22

> hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue. Let your LLM have a fixed input context length of one byte. Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". Define your computa…

> There is nothing stopping inductively increasing the size of the input context and the number of entries in the ground truth table arbitrarily This isn’t induction. You’ve only done the base case, not the induction hypothesis or induction step. Maybe you’ve done those steps in your head but that’s not really a trivial proof as you claim.

Induction is "if this is possible for value X, then it is also possible for value X+1".

Where X isn't used as part of the step this is always true. Nothing I did depends on the size of either the input nor the truth table, so long as both are finite-size and so long as the truth table can be expressed as a function of the input.

An LLM is an arbitrary convolution of the input text; for any mapping, some function you can call an "LLM" produces that function.

Re: Hallucination is inevitable: An innate limitation of large language models

#158

Post by wredue »

golol wrote: post #41
Sparkyte wrote: post #29

Definitely a given, it isn't like AI has an actual brain capable of resolving and forming new connections. The LLM and human brains is that LLMs are interactive compendiums and our brains organize and sort information that ensures survival as an organism. There is no survival of whether or not LLMs are accurate and a machine wouldn't understand what is good or bad without weighted context. Its good for analyze, proce…

The result in the theorem applies to your brain. Your brain can be modelled as an LLM in the sense of the paper up to arbitrarily small error. The result is a diagonalization argument that is not very relevant for the real world.

>your brain can be modelled as an LLM in the sense of the paper

The vast majority of people actually writing LLMs don’t claim this, and in fact, actually claim the very opposite: that LLMs do not accurately model a human brain in any capacity.

The fact is that science has no clue what happens in the nucleus of a neuron, so claiming that computer scientists must is… well. You fill in the word.

Re: Hallucination is inevitable: An innate limitation of large language models

#159

Post by somethingsaid »

t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

I also wonder if having a hallucination-free LLM is even required for it to be useful. Humans can and will hallucinate (by this I mean make false statements in full confidence, not drugs or mental states) and they’re entrusted with all sorts of responsibilities. Humans are also susceptible to illusions and misdirection just like LLMs. So in all likelihood there is simply some state of ‘good enough’ that is satisfactory for most tasks. Perusing the elimination of hallucinations to the nth degree may be a fools errand.

Re: Hallucination is inevitable: An innate limitation of large language models

#160

Post by t_mann »

andsoitis wrote: post #132
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

> the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). To answer "I don't know" requires one to know when you know. To know when you know in turn requires understanding.

Maybe it requires understanding, maybe there are other ways to get to 'I don't know'. There was a paper posted on HN a few weeks ago that tested LLMs on medical exams, and one interesting thing that they found was that on questions where the LLM was wrong (confidently, as usual), the answer was highly volatile with respect to some prompt or temperature or other parameters. So this might show a way for getting to 'I don't know' by just comparing the answers over a few slightly fuzzied prompt variations, and just ask it to create an 'I don't know' answer (maybe with a summary of the various responses) if they differ too much. This is more of a crutch, I'll admit, arguably the LLM (or neither of the experts, or however you set it up concretely) hasn't learnt to say 'I don't know', but it might be a good enough solution in practice. And maybe you can then use that setup to generate training examples to teach 'I don't know' to an actual model (so basically fine-tuning a model to learn its own knowledge boundary).
Post reply on HN