Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

71–80 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#71

Post by ddalex »

cornholio wrote: post #63
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

Actually it seems to me that they do... I asked via custom prompts the various GPTs to give me scores for accuracy, precision and confidence for its answer (in range 0-1), and then I instructed them to stop generating when they feel the scores will be under .9, which seems to pretty much stop the hallucination. I added this as a suffix to my queries.

Re: Hallucination is inevitable: An innate limitation of large language models

#72

Post by amarant »

cubefox wrote: post #55
timini wrote: post #45

Earlier quoted context omitted.

I think its fairly simple, it needs a certain level of proof e.g references to authoritative sources, if not say "i don't know".

Then it is nothing more than a summarizer for search engine results.

A lot of people have said chat-gpt/copilot is a lot like having a robotic junior dev around.

I think perhaps your description is more succinct

Re: Hallucination is inevitable: An innate limitation of large language models

#73

Post by drdrek »

t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

Not saying anything about LLM But in CS in general many issues "cannot be solved" or "Cannot be solved in reasonable time (NP)" but approximations upper bound by some value are solvable in reasonable time (P).

And in the real world if the truck route of amazon is 20% off the mathematically optimal solution the traveling salesman is "Solved" in a good enough way.

Re: Hallucination is inevitable: An innate limitation of large language models

#74

Post by devjab »

LightBug1 wrote: post #35
carlossouza wrote: post #31

Someone smart once said: If it is good, we call it "creativity." If it is bad, we call it "hallucination." This isn't a bug (or limitation, as the authors say). It's a feature.

I imagine the gold is in knowing whether the LLM understands when it's doing either? Isn't this the difference between a human and an LLM? A human knows it's making an educated guess and (should) say so. Or it knows when it's being creative, and can say so. If it doesn't know which is which, then it really does bring it home that LLM's are not that much more than (very sophisticated) mechanical input-output machines.

Isn’t it always hallucinating though? We just don’t care when it gets it “right”. As I understand it, it’s still just probability based on what is likely to be a good set of words to answer the prompt tasking it. It doesn’t actually know anything, it’s just extremely good at making stuff up.

Which is still very useful for a lot of things. Just maybe not things to which value is assigned based on how efficient and correct the answer is. Like you can have GPT make a marketing campaign for you, or you can have it design all the icons you need for your application UI, but you can’t reliably make it wrote high performance back-end code without having humans judge the results. Similarly you can’t use it to teach anyone anything, not really, because unless you’re already an expert on the subject being taught, you aren’t likely to spot when it gets things wrong. I guess you can argue that a lot of teaching is flawed like that, and you wouldn’t be wrong. Like, I was taught that the pyramids was build by slave labour, even after the archeological evidence had shown this to be likely false. But our text books were a decade old because our school didn’t really renew them very often… in such a case GPT might have been a more correct teacher, but the trick is that you won’t really know. Which is made even more complicated by the fact that it might teach different things to different students. Like, I just asked ChatGPT 3.5 who build the pyramids in 3 different prompts, in one it told me it was ordinary people. In the others it told me it was mostly skilled labour under guidance of “architects” and “engineers”. Still better than teaching us it was done by slave labour like my old book, but the book was still consistent in what was considered to be the truth at the time.

Re: Hallucination is inevitable: An innate limitation of large language models

#75

Post by gardenhedge »

tgv wrote: post #52
gardenhedge wrote: post #38

Earlier quoted context omitted.

I don't think so. I think it's asking it to repeat code it has been trained on

Search for a piece of code you wrote. If it's more than 15 lines and not boilerplate, chances are you won't find it anywhere on the net.

To be honest, I imagine I would. Variables named could be different but it would largely be the same as code others have written. For example, I am creating an app in React Native at the moment. My app will not be unique in terms of code but instead in business domain.

Re: Hallucination is inevitable: An innate limitation of large language models

#76

Post by Scene_Cast2 »

I don't think anyone has mentioned Bayesian Neural Nets (I forget the exact term). Sure, the paradigm adds an order of magnitude overhead (at least - and that's why I've never seen it used in the industry), but you can bolt it on to existing architectures.

The basic idea is that besides the probabilities, the network also spits out confidence (IIRC based on how out-of-distribution the input is). There's been a ton of work on getting confidence values out of existing neural nets without as much overhead, but I've never seen those approaches replicate in the industry.

Re: Hallucination is inevitable: An innate limitation of large language models

#77

Post by smusamashah »

ddalex wrote: post #71
cornholio wrote: post #63

Earlier quoted context omitted.

Transformers have no capacity for self reflection, for reasoning about their reasoning process, they don't "know" that they don't know. My interpretation of the paper is that it claims this weakness if fundamental, you can train the network to act as if it knows its knowledge limits, but there will always be an impossible to cover gap for any real world implementation.

Actually it seems to me that they do... I asked via custom prompts the various GPTs to give me scores for accuracy, precision and confidence for its answer (in range 0-1), and then I instructed them to stop generating when they feel the scores will be under .9, which seems to pretty much stop the hallucination. I added this as a suffix to my queries.

Any examples?

Re: Hallucination is inevitable: An innate limitation of large language models

#78

Post by somewhereoutth »

For production systems the considerations are:

- how often does it error?

- how bad are the errors?

- how tolerable are the errors?

- how detectable are the errors?

- how recoverable are the errors?

For example, a pocket calculator that occasionally was out at the 3rd decimal place might do much more damage than one that quite regularly returned NaN.

LLMs error both regularly and badly, so detectability and recoverability are going to be crucial for useful deployment in systems outside of those that have a high tolerance for errors (e.g. algorithmic feeds).

Re: Hallucination is inevitable: An innate limitation of large language models

#79

Post by somewhereoutth »

Seb-C wrote: post #59
carlossouza wrote: post #31

Someone smart once said: If it is good, we call it "creativity." If it is bad, we call it "hallucination." This isn't a bug (or limitation, as the authors say). It's a feature.

That is correct, it's always hallucinating and making things up. Just because those hallucinations sometimes randomly happens to be right, people concluded that being wrong is the exception, while being right is somehow the rule. It's like when people read [insert millenias old text here], finds a part that happens to illustrate something in their life today and conclude that it is a prophecy that predicted the futur…

For heavy LLM users, there is probably a dopamine hit when it does something right, much as gamblers get a hit when the fruit machine pays out. Perhaps LLM use is no more productive than gambling, and perhaps can be abused in a similar way.

Re: Hallucination is inevitable: An innate limitation of large language models

#80

Post by anonylizard »

bbor wrote: post #40

I’m sorry… does this paper just point out that LLMs by definition are not as good at holding data as a direct database? Cause A) duh and b) who cares, they’re intuitive language transformers, not knowledge models. Maybe I’m missing something obvious? This seems like someone torturing math to imply outlandish conclusions that fit their (in this case anti-“AI”) agenda.

It at least disproves LLMs from being 'god models'. They will never be able to solve every problem perfectly.
Post reply on HN