Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

211–220 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#211

Post by andsoitis »

ninetyninenine wrote: post #192
Karellen wrote: post #148

Earlier quoted context omitted.

They generate text which looks like the kind of text that people who do have understanding generate.

Two key things here to realize. People also often don't understand things and have trouble separating fact from fiction. By logic only one religion or no religion is true. Consequently also by logic most religions in the world where their followers believe the religion to be true are hallucinating. The second thing to realize that your argument doesn't really apply. Its in theory possible to create a stochastic parro…

> People also often don't understand things and have trouble separating fact from fiction.

That's not the point being argued. Understanding, critical thinking, knowledge, common sense, etc. all these things exist on a spectrum - both in principle and certainly in humans. In fact, in any particular human there are different levels of competence across these dimensions.

What we are debating, is whether or not, an LLM can have understanding itself. One test is: can an LLM understand understanding? The human mind has come to the remarkable understanding that understanding itself is provisional and incomplete.

Re: Hallucination is inevitable: An innate limitation of large language models

#212

Post by beardedwizard »

ninetyninenine wrote: post #182

Fiction and story writing is hallucination. It is the opposite of a stochastic parrot. We've achieved both extremes of AI. Computers can be both logical machines and hallucinators. Our goal is to create a machine that can be both at the same time and can differentiate between both. That's the key. Hallucination is important but the key is for the computer to be self aware about when it's hallucinating. Of course it's…

It is not the opposite of stochastic parrot, it is exactly the same thing only the predictions are worse due to sparse training data.

Re: Hallucination is inevitable: An innate limitation of large language models

#213

Post by [deleted] »

username332211 wrote: post #195
caditinpiscinam wrote: post #180

Earlier quoted context omitted.

In real world conversations, people are constantly saying "I don't know"; but that doesn't really happen online. If you're on reddit or stack overflow or hacker news and you see a question you don't know the answer to, you normally just don't say anything. If LLMs are being trained on conversations pulled from the internet then they're missing out on a ton of uncertain responses. Maybe LLMs don't truly "understand" q…

If they were trained on more uncertain content, what happens if the most probable answer to a question is "I don't know", even though an answer exists in it's training set? Suppose 99.3% of answers to 'What is the airspeed velocity of an unladen swallow?" are "I don't know that." and the remainder are "11 m/s". What would the model answer? When the LLM answers "I don't know.", this could be a hallucination just as ea…

[deleted]

Re: Hallucination is inevitable: An innate limitation of large language models

#214

Post by beardedwizard »

fl7305 wrote: post #204
caditinpiscinam wrote: post #180

Earlier quoted context omitted.

In real world conversations, people are constantly saying "I don't know"; but that doesn't really happen online. If you're on reddit or stack overflow or hacker news and you see a question you don't know the answer to, you normally just don't say anything. If LLMs are being trained on conversations pulled from the internet then they're missing out on a ton of uncertain responses. Maybe LLMs don't truly "understand" q…

If you ask ChatGPT a question, and tell it to either respond with the answer or "I don't know", it will respond "I don't know" if you ask it whether you have a brother or not.

This has nothing to do with thinking and everything to do with the fact that given that input the answer was the most probable output given the training data.

Re: Hallucination is inevitable: An innate limitation of large language models

#215

Post by beardedwizard »

DiogenesKynikos wrote: post #200
sandworm101 wrote: post #163

Earlier quoted context omitted.

They cannot say "I dont know" because they dont actually know anything. The answers are not comming from a thinking mind but a complex pattern-fitting supercomputer hovering over a massive table of precomputed patterns. It computes your input then looks to those patterns and spits out the best match. There is no thinking brain with a conceptual understanding of its own limitations. Getting an "i dont know" from curre…

> The answers are not comming from a thinking mind but a complex pattern-fitting supercomputer hovering over a massive table of precomputed patterns. Are you sure you're not also describing the human brain? At some point, after we have sufficiently demystified the workings of the human brain, it will probably also sound something like, "Well, the brain is just a large machine that does X, Y and Z [insert banal-soundi…

Human brains form new connections dynamically. Llms are trained on connections human brains have already made. They never make new connections that aren't in training data.

Nothing was synthesized, all the data was seen before and related to each other by vector similarity.

It can just parrot the collective understanding humans already have and teach it.

Re: Hallucination is inevitable: An innate limitation of large language models

#216

Post by andsoitis »

t_mann wrote: post #176
andsoitis wrote: post #169

Earlier quoted context omitted.

> Maybe it requires understanding, maybe there are other ways to get to 'I don't know'. > This is more of a crutch, I'll admit, arguably the LLM (or neither of the experts, or however you set it up concretely) hasn't learnt to say 'I don't know', but it might be a good enough solution in practice. And maybe you can then use that setup to generate training examples to teach 'I don't know' to an actual model (so basica…

Ok, but LLMs are just tools, and I'm just asking how a tool can be made more useful. It doesn't really matter why an LLM tells you to go look elsewhere, it's simply more useful if it does than if it hallucinates. And usefulness isn't binary, getting the error rate down is also an improvement.

> Ok, but LLMs are just tools, and I'm just asking how a tool can be made more useful.

I think I know what you're after (notice my self-awareness to qualify what I say I know): that the tool's output can be relied upon without applying layers of human judgement (critical thinking, logical reasoning, common sense, skepticism, expert knowledge, wisdom, etc.)

There are a number of boulders in that path of clarity. One of the most obvious boulders is that for an LLM the inputs and patterns that act on the input are themselves not guaranteed to be infallible. Not only in practive, but also in principle: the human mind (notice this expression doesn't refer to a thing you can point to) has come to understand that understanding is provisional, incomplete, a process.

So while I agree with you that we can and should improve the accuracy of the output of these tools given assumptions we make about the tools humans use to prove facts about the world, you will always want to apply judgment, skepticism, critical thinking, logical evaluation, intuition, etc. depending on the risk/reward tradeoff of the topic you're relying on the LLM for.

Re: Hallucination is inevitable: An innate limitation of large language models

#217

Post by beardedwizard »

ninetyninenine wrote: post #188
mannykannot wrote: post #155

Earlier quoted context omitted.

This is a fair question: LLMs do challenge the easy assumption (as made, for example, in Searle's "Chinese Room" thought experiment) that computers cannot possibly understand things. Here, however, I would say that if an LLM can be said to have understanding or knowledge of something, it is of the patterns of token occurrences to be found in the use of language. It is not clear that this also grants the LLM any under…

Explain sora. It must have of course a blurry understanding of reality to even produce those videos. I think we are way past the point of debate here. LLMs are not stochastic parrots. LLMs do understand an aspect of reality. Even the LLMs that are weaker than sora understand things. What is debatable is whether LLMs are conscious. But whether it can understand something is a pretty clear yes. But does it understand e…

I do not understand these comments at all. Sora was trained on billions of frames from video and images - they were tagged with words like "ballistic missile launch" and "cinematic shot" and it simply predicts the pixels like every other model. It stores what we showed it, and reproduces it when we ask - this has nothing to do with understanding and everything to do with parroting. The fact that it's now a stream of images instead of just 1 changes nothing about it.

Re: Hallucination is inevitable: An innate limitation of large language models

#218

Post by barnacs »

phh wrote: post #147
keepamovin wrote: post #103

This is why you need to pair language learning with real world experience. These robots need to be given a world to explore -- even a virtual one -- and have consequences within, and to survive it. Otherwise it's all unrooted sign and symbol systems untethered to experience.

I think I agree with you (I even upvoted), but this might be an anthropomorphism. Back like 3-5 years ago, we already thought that about LLMs: They couldn't answer questions about what would fall when stuff are attached together in some non-obvious way, and the argument back then was that you had to /experience/ it to realize it. But LLMs have long fixed those kind of issues. The way LLMs "resolve" questions is very…

Think of it this way:

Intelligent beings in the real world have a very complex built-in biological error function rooted in real world experiences: sensory inputs, feelings, physical and temporal limitations and so on. You feel pain, joy, fear, have a limited lifetime, etc.

"AI" on the other hand only have an external error function, usually roughly designed to minimize the difference of the output from that of an actually intelligent real world being.

Re: Hallucination is inevitable: An innate limitation of large language models

#219

Post by someplaceguy »

Fripplebubby wrote: post #177

The core argument in this paper it seems to me from scanning it is that because P != NP therefore LLMs will hallucinate answers to NP-complete problems. I think this is a clever point and an interesting philosophical question (about math, computer science, and language), but I think people are mostly trying to apply this using our commonsense notions of "LLM hallucination" rather than the formal notion they use in th…

> because P != NP therefore LLMs will hallucinate answers to NP-complete problems.

I haven't read the paper, but that sounds like it would only be true if the definition of "hallucinating" is giving a wrong answer, but that's not how it's commonly understood.

When people refer to LLMs hallucinating, they are indeed referring to an LLM giving a wrong (and confident) answer. However, not all wrong answers are hallucinations.

An LLM could answer "I don't know" when asked whether a certain program halts and yet you wouldn't call that hallucinating. However, it sounds like the paper authors would consider "I don't know" to be a hallucinating answer, if their argument is that LLMs can't always correctly solve an NP-complete problem. But again, I haven't read the paper.

Re: Hallucination is inevitable: An innate limitation of large language models

#220

Post by ben_w »

mistermann wrote: post #127
ben_w wrote: post #105

Earlier quoted context omitted.

We can do logical reasoning, but we're very bad at it and often take shortcuts either via pattern matching, memory, or "common sense". Baseball and bat together cost $1.10, the bat is $1 more than the ball, how much does the ball cost? A French plane filled with Spanish passengers crashes over Italy, where are the survivors buried? An armed man enters a store, tells the cashier to hand over the money, and when he dep…

Humans also have various culturally flavored, implicit "you know what I mean" algorithms on each end to smooth out "irrelevant" misunderstandings and ensure a cordial interaction, a cultural prime directive.

Sure. I think LLMs are good at that kind of thing.

My final example demonstrates how those cultural norms cause errors, it was from a logical thinking session at university, where none of the rest of my group could accept my (correct) claim that the answer was "not enough information to answer" even when I gave a (different but also plausible) non-robbery scenario and pointed out that we were in a logical thinking training session which would have trick questions.

My dad had a similar anecdote about not being able to convince others of the true right answer, but his training session had the setup "you crash landed on the moon, here's a list of stuff in your pod, make an ordered list of what you take with you to reach a survival station", and the correct answer was 1. oxygen tanks, 2. a rowing boat, 3. everything else, because the boat is a convenient container for everything else and you can drag it along the surface even though there's no water.

Post reply on HN