Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

141–150 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#141

Post by [deleted] »

ddalex wrote: post #71
cornholio wrote: post #63

Earlier quoted context omitted.

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

Actually it seems to me that they do... I asked via custom prompts the various GPTs to give me scores for accuracy, precision and confidence for its answer (in range 0-1), and then I instructed them to stop generating when they feel the scores will be under .9, which seems to pretty much stop the hallucination. I added this as a suffix to my queries.

[deleted]

Re: Hallucination is inevitable: An innate limitation of large language models

#142

Post by gessha »

nottorp wrote: post #87
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

> we can get LLMs to answer 'I don't know' more often Have any nets been trained specifically to be able to go to an 'i don't know' state, I wonder? It may be the humans' fault.

Yes, you can find some of the work on this topic under the terms open world recognition or open world X where X is a topic in computer vision or NLP.

https://arxiv.org/abs/2011.12906

Re: Hallucination is inevitable: An innate limitation of large language models

#143

Post by sorokod »

ddalex wrote: post #71
cornholio wrote: post #63

Earlier quoted context omitted.

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

Actually it seems to me that they do... I asked via custom prompts the various GPTs to give me scores for accuracy, precision and confidence for its answer (in range 0-1), and then I instructed them to stop generating when they feel the scores will be under .9, which seems to pretty much stop the hallucination. I added this as a suffix to my queries.

You are ok with not defining what each of your attributes means but willing to accept that:

the LLM will read your mind and correctly interpret them in the context of its internal state

the LLM will calculate non hallucinated values

Re: Hallucination is inevitable: An innate limitation of large language models

#144

Post by Borealid »

resolutebat wrote: post #25
Borealid wrote: post #22

> hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue. Let your LLM have a fixed input context length of one byte. Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". Define your computa…

> Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". This is basically the same as saying "train your LLM until they never hallucinate", which reduces your claim to a tautology: an LLM trained not to hallucinate does not hallucinate. The trick is making that happen.

It's a tautology that for a given truth table mapping inputs to "correct" outputs there exists a function that produces that mapping.

Saying that you can't train an LLM to NOT hallucinate is saying that it's impossible for any LLM to always produce output matching any particular truth table.

There may exist truth tables where it's not possible to produce an LLM to match them (for some finite size of LLM perhaps), but my claim isn't a tautology - it's just an assertion that there exist some truth tables which an LLM can be trained to match.

It may be tricky to make the LLM in the first place, but it's certainly not as tricky to verify it. You can test it with every input you consider to be present in the truth table, record its results, and throw it away if it doesn't match. The only possible results here are either non-convergeance or a "perfect" LLM.

You can get rid of the non-convergeance if you bound the size of the LLM and literally iteratively verify every single possible model. The authors of the paper didn't do that. For trivially sized models and inputs, that's completely possible. For a 7B parameter model, nobody is doing that ever. But you can prove the title statement wrong with a ten-parameter model and a one-byte input.

Re: Hallucination is inevitable: An innate limitation of large language models

#145

Post by gessha »

intended wrote: post #117
gardenhedge wrote: post #38

Earlier quoted context omitted.

I don't think so. I think it's asking it to repeat code it has been trained on

The term “Hallucinate” is a misnomer. Humans can hallucinate, we can get sick and perceive a world which is incongruous with reality. LLMs are just generating tokens. Hallucination perpetuates an unhelpful anthropomorphization of LLMs.

I don’t think the term is that bad here because I haven’t seen a lot of comparisons with human hallucinations.

Users see it as a machine artifact.

Re: Hallucination is inevitable: An innate limitation of large language models

#146

Post by Borealid »

less_less wrote: post #66
Borealid wrote: post #22

> hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue. Let your LLM have a fixed input context length of one byte. Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". Define your computa…

I also disagree with the paper, but not for the same reason. > With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue. Unsurprisingly, that one sentence fragment doesn't capture the entirety of their assumptions. Instead they prove something intuitively obvious, along the lines of: LLMs with arbitrary-length inputs and certain resource restrictions (e.g. they ca…

It's not a bad paper honestly, I just don't like it when people take a line from it and assume something untrue.

The pigeonhole principle proves that if you only have N slots to work with, and you need to fit N+1 items into them, you're going to get at least one slot with at least two items. That makes sense, and it logically follows that constrained functions can't perfectly mirror less-constrained ones: at some point a "wrong" and a "right" input have to produce the same output.

Re: Hallucination is inevitable: An innate limitation of large language models

#147

Post by phh »

keepamovin wrote: post #103

This is why you need to pair language learning with real world experience. These robots need to be given a world to explore -- even a virtual one -- and have consequences within, and to survive it. Otherwise it's all unrooted sign and symbol systems untethered to experience.

I think I agree with you (I even upvoted), but this might be an anthropomorphism.

Back like 3-5 years ago, we already thought that about LLMs: They couldn't answer questions about what would fall when stuff are attached together in some non-obvious way, and the argument back then was that you had to /experience/ it to realize it. But LLMs have long fixed those kind of issues.

The way LLMs "resolve" questions is very different from us. At this point, I think that if we want to prove that LLMs need to be rooted in the real world to achieve intelligence, we need to find some real-world phenomenon that is so obvious that noone ever wrote about it... but then we'd have written about it?

Re: Hallucination is inevitable: An innate limitation of large language models

#148

Post by Karellen »

MuffinFlavored wrote: post #139
andsoitis wrote: post #132

Earlier quoted context omitted.

> the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). To answer "I don't know" requires one to know when you know. To know when you know in turn requires understanding.

how did LLMs get this far without any concept of understanding? how much further can they go until they become “close enough”?

They generate text which looks like the kind of text that people who do have understanding generate.

Re: Hallucination is inevitable: An innate limitation of large language models

#149

Post by gessha »

tgv wrote: post #52
gardenhedge wrote: post #38

Earlier quoted context omitted.

I don't think so. I think it's asking it to repeat code it has been trained on

Search for a piece of code you wrote. If it's more than 15 lines and not boilerplate, chances are you won't find it anywhere on the net.

This reminds me of a bit from a Hugh and Laurie sketch:

> Imagine a piano keyboard, eighty-eight keys, only eighty-eight and yet, and yet, new tunes, melodies, harmonies are being composed upon hundreds of keyboards every day in Dorset alone. Our language, Tiger, our language, hundreds of thousands of available words, frillions of possible legitimate new ideas, so that I can say this sentence and be confident it has never been uttered before in the history of human communication: "Hold the newsreader's nose squarely, waiter, or friendly milk will countermand my trousers." One sentence, common words, but never before placed in that order. And yet, oh and yet, all of us spend our days saying the same things to each other, time after weary time, living by clichaic, learned response: "I love you", "Don't go in there", "You have no right to say that", "shut up", "I'm hungry", "that hurt", "why should I?", "it's not my fault", "help", "Marjorie is dead". You see? That surely is a thought to take out for a cream tea on a rainy Sunday afternoon.

https://abitoffryandlaurie.co.uk/sketches/language_conversat...

Re: Hallucination is inevitable: An innate limitation of large language models

#150

Post by viktour19 »

If hallucination is inevitable? What should developers do?

Design user experiences that align users with this behaviour!

Relatedly, I built a game to demonstrate how one might calibrate users to the responses of LLMs:

https://news.ycombinator.com/item?id=39255583

Post reply on HN