Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

251–260 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#251

Post by kalkin »

thargor90 wrote: post #241
fauigerzigerk wrote: post #237

Earlier quoted context omitted.

Does the paper distinguish between hallucination and approximation? If LLMs could be trained to approximate NP-complete functions rather than making stuff up, that would be good enough in many contexts. I guess it's what humans would do.

You cannot approximate NP-complete functions. If you could approximate them with a practically useful limited error and at most P effort you would have solved P=NP. (disclaimer my computer science classes have been a long time ago)

This isn't correct. What you may be remembering is that some (not all) NP complete problems have limits on how accurately they can be approximated (unless P = NP). But approximation algorithms for NP complete problems form a whole subfield of CS.

Re: Hallucination is inevitable: An innate limitation of large language models

#252

Post by c22 »

tsimionescu wrote: post #226
cpuguy83 wrote: post #190

Earlier quoted context omitted.

Are humans not the same way? There's a saying "I don't know what I don't know".

The existence of such a saying means little: it is uncontroversially true that humans often do know what they don't know. For example, I know for sure that I don't know how long I will live. This disproves the saying.

The saying isn't implying that there aren't any things you know you don't know, it's saying that some of the things you don't know are not even conceivable to you without knowing more.

You know the question "how long will I live?" And you know this question has an answer which is unknown to you. But there also exist other questions that you don't even know to ask.

Re: Hallucination is inevitable: An innate limitation of large language models

#253

Post by fauigerzigerk »

thargor90 wrote: post #241
fauigerzigerk wrote: post #237

Earlier quoted context omitted.

Does the paper distinguish between hallucination and approximation? If LLMs could be trained to approximate NP-complete functions rather than making stuff up, that would be good enough in many contexts. I guess it's what humans would do.

You cannot approximate NP-complete functions. If you could approximate them with a practically useful limited error and at most P effort you would have solved P=NP. (disclaimer my computer science classes have been a long time ago)

Perhaps I'm not using the vocabulary correctly here.

What I mean is, if you ask a human to solve a travelling salesman problem and they find it too hard to solve exactly, they will still be able to come up with a better than average solution. This is what I called approximation (but maybe this is incorrect?).

Hallucination would be to choose a random solution and claim that it's the optimum.

Re: Hallucination is inevitable: An innate limitation of large language models

#254

Post by renegade-otter »

p1esk wrote: post #208
Fripplebubby wrote: post #177

The core argument in this paper it seems to me from scanning it is that because P != NP therefore LLMs will hallucinate answers to NP-complete problems. I think this is a clever point and an interesting philosophical question (about math, computer science, and language), but I think people are mostly trying to apply this using our commonsense notions of "LLM hallucination" rather than the formal notion they use in th…

The only way to reduce hallucinations in both humans and LLMs is to increase their general intelligence and their knowledge of the world.

It's statistical prediction. LLMs do not "understand" the world by definition.

Ask an image generator to make "an image of a woman sitting on a bus and reading a book".

Images will be either a horror show or at best full of weird details that do not match the real world - because it's not how any of this works.

It's a glorified auto-complete that only works due to the massive amounts of data it is trained on. Throw in any complex interactions it has not seen in the training data and it's all over.

Re: Hallucination is inevitable: An innate limitation of large language models

#255

Post by rf15 »

t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

> I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often

I feel like writing an entire paper about the practical approach to the problems posed in this paper, but you'll probably have to first formally define the language used in the training data before you can try to map it (through training and sampling algos, which this paper conveniently skipped) to the target form. This sounds really fun at first, but then we're once again talking about the strict formalisation of natural language (which you could still do - the training data is limited and fixed!)

Re: Hallucination is inevitable: An innate limitation of large language models

#257

Post by calf »

Borealid wrote: post #146
less_less wrote: post #66

Earlier quoted context omitted.

I also disagree with the paper, but not for the same reason. > With this definition, you can trivially prove the titular sentence - "hallucination is inevitable" - is untrue. Unsurprisingly, that one sentence fragment doesn't capture the entirety of their assumptions. Instead they prove something intuitively obvious, along the lines of: LLMs with arbitrary-length inputs and certain resource restrictions (e.g. they ca…

It's not a bad paper honestly, I just don't like it when people take a line from it and assume something untrue. The pigeonhole principle proves that if you only have N slots to work with, and you need to fit N+1 items into them, you're going to get at least one slot with at least two items. That makes sense, and it logically follows that constrained functions can't perfectly mirror less-constrained ones: at some poi…

So is it saying LLMs have polynomial running time and that's it? LLMs can't solve SAT properly because of running time argument?

Re: Hallucination is inevitable: An innate limitation of large language models

#258

Post by [deleted] »

renegade-otter wrote: post #254
p1esk wrote: post #208

Earlier quoted context omitted.

The only way to reduce hallucinations in both humans and LLMs is to increase their general intelligence and their knowledge of the world.

It's statistical prediction. LLMs do not "understand" the world by definition. Ask an image generator to make "an image of a woman sitting on a bus and reading a book". Images will be either a horror show or at best full of weird details that do not match the real world - because it's not how any of this works. It's a glorified auto-complete that only works due to the massive amounts of data it is trained on. Throw i…

[deleted]

Re: Hallucination is inevitable: An innate limitation of large language models

#259

Post by wseqyrku »

intended wrote: post #112
t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

If a model can say ‘I don’t know’, then the hallucination problem would also be solved. When we say “know” it usually means being factual. For an LLM to ‘know’ it doesn’t know, it would have had to move away from pure correlations on words, and meta processing about its own results. I can see this happen with two LLMs working together (and there are Evals that use just this), however each LLM still has no self awaren…

Feeding the output to another inference would probably reduce hallucination? but I have this impression that the models talking to each other don't need to know English, a byte stream would do. Just like the hidden layers of an ANN.

Re: Hallucination is inevitable: An innate limitation of large language models

#260

Post by baq »

timr wrote: post #51
baq wrote: post #4

Earlier quoted context omitted.

This again. They’re trained to generate probable text. The mechanisms created in the parameter blob during training to do that are basically a mystery and have to be pulled out of the model with digital brain surgery. E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those paramete…

> E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters. Broadly agreed, but there's no "representation"...the model has no memory, let alone a "concept" of a chessboard. It's just trained on a bunch of textual replays of chess games, and this works well enough for a g…

The fact that tweaking parameters which appear to store the board makes it play according to the tweaked numbers instead of what was passed to it the context (i.e. working memory) directly contradicts your assertion that LLMs have no memory. The context is their memory.

I can’t comment on your drug generation task - they aren’t magic, if the training didn’t result in a working drug model in the billions of params you’ll get garbage output, not very surprising.

My point boils down to the ‘what’s more likely’ question: magical stochastic parrots which just by accident manage to create understandable and coherent responses to unreasonably large set of questions or the magic is actually some kind of a world model, or multiple, inside the blob of numbers, outputs of which steer the probabilities, just as this confirmed chess example. My bet is on learned world models because I’m not convinced there’s magic in our physical world.

Post reply on HN