Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

301–310 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#301

Post by Animats »

Fripplebubby wrote: post #177

The core argument in this paper it seems to me from scanning it is that because P != NP therefore LLMs will hallucinate answers to NP-complete problems. I think this is a clever point and an interesting philosophical question (about math, computer science, and language), but I think people are mostly trying to apply this using our commonsense notions of "LLM hallucination" rather than the formal notion they use in th…

Yes. It looks like they introduce infinities and then run into the halting problem for infinities. That may not be helpful.

The place where this argument gets into trouble is where it says "we define hallucination in a formal world where all we care about is a computable ground truth function f on S." This demands a reliable, computable predicate for truth. That alone is probably not possible.

If, however, we are willing to accept a ground truth function with outputs

    - True
    - False
    - Unknown
    - Resource limit exceeded
that problem can be avoided. Now the goal is manageable - return True or False only when those results are valid, and try to reduce the fraction of useful queries for which Unknown and Resource Limit Exceeded are returned.

The same problem comes up in program verification systems, and has been dealt with in the same way for decades. Sometimes, deciding if something is true is too much work.

Re: Hallucination is inevitable: An innate limitation of large language models

#302

Post by username332211 »

patmcc wrote: post #291
username332211 wrote: post #195

Earlier quoted context omitted.

If they were trained on more uncertain content, what happens if the most probable answer to a question is "I don't know", even though an answer exists in it's training set? Suppose 99.3% of answers to 'What is the airspeed velocity of an unladen swallow?" are "I don't know that." and the remainder are "11 m/s". What would the model answer? When the LLM answers "I don't know.", this could be a hallucination just as ea…

Right, but "I don't know" is a pretty safe hallucination (if it is one). My main worry about hallucinations is it means I absolutely can't rely on the output for anything important. If I ask what the safe dose for Tylenol for an infant is, the answer needs to be either correct or "I don't know". It's not acceptable for it to hallucinate 10x the safe dose.

The thing is, if you answer "I don't know" based on statistics, you end up creating a sliding scale of sorts. You get some measure of an increase in safety, but the model is less useful.

Currently,we have models that make stuff up when they don't know the answer. On the other end, we'd have a model that's refuses to answer any question that's not common knowledge. It'll be safe (though it can never be completely safe), but essentially useless.

I suspect it'll be impossible to make a completely trustworthy and useful model unless it somehow has a concept of it's own knowledge. And can you have a concept of one's knowledge if you lack a concept of self?

Re: Hallucination is inevitable: An innate limitation of large language models

#303

Post by dragonwriter »

someplaceguy wrote: post #261
dragonwriter wrote: post #249

Earlier quoted context omitted.

> Answering "I don't know" or "I can't answer that" is a perfectly plausible response to a difficult logical problem/question. Sure, and you can train LLMs to produce answers like that more often, but then users will say your model is lazy and doesn't even try, whereas if you train it to be more likely to produce something that looks like a solution more often, people will think “wow, the AI solved this problem I cou…

> Sure, and you can train LLMs to produce answers like that more often, but then users will say your model is lazy and doesn't even try, whereas if you train it to be more likely to produce something that looks like a solution more often, people will think “wow, the AI solved this problem I couldn't solve”. Are you saying that LLMs can't learn to discriminate between which questions they should answer "I don't know"…

> Are you saying that LLMs can't learn to discriminate between which questions they should answer "I don't know" vs which questions they should try to provide an accurate answer?

No, I am saying that they are specific trained to do that, and that the results seen in practice on common real-world LLMs reflect the bias of the specific training they are given for providing concrete answers.

> Sure, there will be an error rate, but surely you can train an LLM to minimize it?

Giving some answer to a question that cannot be infallibly solved analytically is not necessarily an error. In fact, I would argue that providing useful answers in situations like that is among the motivating use cases for AI.

(Whether or not the answers current LLMs provide in these cases are useful is another question, but you miss 100% of the shots you don’t take.)

Re: Hallucination is inevitable: An innate limitation of large language models

#305

Post by p1esk »

lazide wrote: post #236
FpUser wrote: post #222

Earlier quoted context omitted.

You post amounts to: in order to be smarter I need to increase my smartness. Great insight.

I think it's more subtly misleading - to be smarter, I need more knowledge. But knowledge != smart, knowledge == informed, or educated. And the problem is more - how can an LLM tell us it doesn't know something instead of just making up good sounding, but completely delusional answers. Which arguably isn't about being smart, and is only tangentially about less or more (external) knowledge really. It's about self-know…

Hallucinations are an interesting problem - in both humans and statistical models. If we asked an average person 500 years ago how the universe works, they would have confidently told you the earth is flat and it rests on a giant turtle (or something like that). And that there are very specific creatures - angels and demons who meddle in human affairs. And a whole a lot more which has no grounding in reality.

How did we manage to reduce that type of hallucination?

Re: Hallucination is inevitable: An innate limitation of large language models

#306

Post by HKH2 »

somewhereoutth wrote: post #285
HKH2 wrote: post #104

Earlier quoted context omitted.

You might not have found any uses that suit your niche, but that doesn't mean those of us who have are just making up stories about productivity.

"You might not have found any games that suit your skill set, but that doesn't mean those of us who have are just making up stories about making money" Sorry, somewhat trite and unfair, but, if there is a gambling-like dopamine reward cycle occurring, then the users would have a hard time being truly objective about any productivity boost in total . They may instead focus on the 'wins', without taking into account an…

Sure, confirmation bias exists, but you can compare with the alternatives.

E.g. a search engine can give you zero useful results, and you can fine tune your query and still get nothing after scrolling through pages of results (Do people really take the losses into account when using search engines?) I find prompt engineering with LLMs more useful because you get nudged in interesting directions, and even if you come away with no direct results, you have more of an idea of what you are looking for. Maybe lateral thinking is overrated.

Re: Hallucination is inevitable: An innate limitation of large language models

#308

Post by jncfhnb »

sandworm101 wrote: post #163
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

They cannot say "I dont know" because they dont actually know anything. The answers are not comming from a thinking mind but a complex pattern-fitting supercomputer hovering over a massive table of precomputed patterns. It computes your input then looks to those patterns and spits out the best match. There is no thinking brain with a conceptual understanding of its own limitations. Getting an "i dont know" from curre…

> They cannot say "I dont know" because they dont actually know anything.

print(“I don’t know”)

You don’t need proper cognition to identify that the answer is not stored in source data. Your conception of the model is incomplete as is easily demonstrable by testing such cases now. Chat gpt does just fine on your simpsons test.

You, however, have made up an answer of how something works that you don’t actually know despite your cognition

Re: Hallucination is inevitable: An innate limitation of large language models

#309

Post by TheBlight »

DinaCoder99 wrote: post #307

"Hallucination" implies perception of non-real things, not generation of phrases that map poorly to reality (or are simply incoherent). It seems like a really bad term for this phenomenon.

"Bullsh***ing" seems more apt.

Re: Hallucination is inevitable: An innate limitation of large language models

#310

Post by nedt »

Well humans believe that vacination either kills people or gives them chips for tracking and the top politicians are lizard people drinking the blood of children kept in caves and they had to fake a pandemic to get them out. I'd say an A.I. hallucinating isn't that far off from real humans. It's rather the recipient that needs to interpret any response from either.
Post reply on HN