Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

81–90 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#81

Post by mepiethree »

Borealid wrote: post #22

> hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue. Let your LLM have a fixed input context length of one byte. Continue training the LLM until such a time as it replies to the input "A" with "yes" and all other inputs with "no". Define your computa…

> There is nothing stopping inductively increasing the size of the input context and the number of entries in the ground truth table arbitrarily

This isn’t induction. You’ve only done the base case, not the induction hypothesis or induction step. Maybe you’ve done those steps in your head but that’s not really a trivial proof as you claim.

Re: Hallucination is inevitable: An innate limitation of large language models

#82

Post by mike_hearn »

cornholio wrote: post #63
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

They do have that capacity. The generated token probabilities reflect some degree of certainty and additionally that information is available earlier in the network too, such that it can explain its own level of confidence.

Re: Hallucination is inevitable: An innate limitation of large language models

#83

Post by mike_hearn »

Certhas wrote: post #50
timini wrote: post #45

Earlier quoted context omitted.

I think its fairly simple, it needs a certain level of proof e.g references to authoritative sources, if not say "i don't know".

LLMs don't have a concept of sources for their statements. Ask them to give you some literature recommendations on something it has explained to you. You'll get plenty of plausible sounding papers that don't exist. Humans know to some extent why they know (read it in a text book, colleague mentioned it). LLMs don't seem to.

Ask a human to provide accurate citations for any random thing they know and they won't be able to do a good job either. They'd probably have to search to find it, even if they know they got it from a document originally and have some clear memory of what it said.

Re: Hallucination is inevitable: An innate limitation of large language models

#84

Post by mysterydip »

cornholio wrote: post #63
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

Would a possible solution be a hybrid approach? I remember back years ago seeing AI research around predicate logic, with hundreds of thousands of classification entries and "x can't be y" type stuff.

Maybe the potential output of an LLM could be run through that kind of logic and fed back to itself for sanity before being decided as final output?

Re: Hallucination is inevitable: An innate limitation of large language models

#85

Post by macrolime »

They define an LLM as "a probabilistic model of a string that conditions the output at time t based on all the tokens that come before it in the string".

I think that definition is wide enough to include human intelligence, so their finding should be equally valid for humans.

Re: Hallucination is inevitable: An innate limitation of large language models

#86

Post by Jensson »

ddalex wrote: post #71
cornholio wrote: post #63

Earlier quoted context omitted.

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

Actually it seems to me that they do... I asked via custom prompts the various GPTs to give me scores for accuracy, precision and confidence for its answer (in range 0-1), and then I instructed them to stop generating when they feel the scores will be under .9, which seems to pretty much stop the hallucination. I added this as a suffix to my queries.

The model will just hallucinate scores, they don't have the ability to self reflect with words like that, there is no function for it to associate the word 90 with its own mental score 90% so anything it will say about those scores is just a hallucination.

Re: Hallucination is inevitable: An innate limitation of large language models

#87

Post by nottorp »

t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

> we can get LLMs to answer 'I don't know' more often

Have any nets been trained specifically to be able to go to an 'i don't know' state, I wonder?

It may be the humans' fault.

Re: Hallucination is inevitable: An innate limitation of large language models

#88

Post by rini17 »

az09mugen wrote: post #37
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

I think there is no easy way to make an LLM answer "I don't know". For that, it should learn among all the stuff ingested when people effectively don't know. But most people on internet write down irrelevant stuff even when they don't know instead of simply writing "I don't know". That's a very good point.

Maybe it needs some memory retrieval step that can measure the confidence - whether there's anything related to the prompt. No idea how to train a LLM to do that.

Re: Hallucination is inevitable: An innate limitation of large language models

#89

Post by nottorp »

Certhas wrote: post #50
timini wrote: post #45

Earlier quoted context omitted.

I think its fairly simple, it needs a certain level of proof e.g references to authoritative sources, if not say "i don't know".

LLMs don't have a concept of sources for their statements. Ask them to give you some literature recommendations on something it has explained to you. You'll get plenty of plausible sounding papers that don't exist. Humans know to some extent why they know (read it in a text book, colleague mentioned it). LLMs don't seem to.

LLMs don't have any concepts period.

Re: Hallucination is inevitable: An innate limitation of large language models

#90

Post by [deleted] »

nottorp wrote: post #87
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

> we can get LLMs to answer 'I don't know' more often Have any nets been trained specifically to be able to go to an 'i don't know' state, I wonder? It may be the humans' fault.

[deleted]
Post reply on HN