Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

401–410 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#401

Post by goatlover »

HDThoreaun wrote: post #270
renegade-otter wrote: post #254

Earlier quoted context omitted.

It's statistical prediction. LLMs do not "understand" the world by definition. Ask an image generator to make "an image of a woman sitting on a bus and reading a book". Images will be either a horror show or at best full of weird details that do not match the real world - because it's not how any of this works. It's a glorified auto-complete that only works due to the massive amounts of data it is trained on. Throw i…

I think the situation is a lot more complicated than youre making it out to be. GPT4 for example can be very good at tasks it has not seen in the training data. The philosophy of mind is much more open ended and less understood than you seem to think.

How do you know what's in the training data? Has OpenAI made the dataset searchable so we can see that GPT4 is performing tasks not in there?

Re: Hallucination is inevitable: An innate limitation of large language models

#402

Post by mr_toad »

samatman wrote: post #325
Fripplebubby wrote: post #177

The core argument in this paper it seems to me from scanning it is that because P != NP therefore LLMs will hallucinate answers to NP-complete problems. I think this is a clever point and an interesting philosophical question (about math, computer science, and language), but I think people are mostly trying to apply this using our commonsense notions of "LLM hallucination" rather than the formal notion they use in th…

Hallucination is a misnomer in LLMs and it depresses me that it has solidified as terminology. When humans do this, we call it confabulation. This is a psychiatric symptom where the sufferer can't tell that they're lying, but fills in the gaps in their knowledge with bullshit which they make up on the spot. Hallucination is an entirely different symptom. And no, confabulation isn't a normal thing which humans do, and…

I suspect it’s just an incomplete memory and no “filter”. LLMs aren’t self-aware enough to judge their own confidence in their responses, so they don’t know when to shut up.

Re: Hallucination is inevitable: An innate limitation of large language models

#403

Post by mr_toad »

flextheruler wrote: post #235
foobarian wrote: post #205

Earlier quoted context omitted.

Well in the example of an NP complete problem, a human might realize they are having trouble coming up with an optimal solution and start analyzing complexity. And once they have a proof might advise you accordingly and perhaps suggest a good enough heuristic.

Is the commenter above you implying humans hallucinate to the level of LLMs? Maybe hungover freshman working on a tight deadline without having read the book do, but not professionals. Even a mediocre employees will often realize they’re stuck, seek assistance, and then learn something from the assistance instead of making stuff up.

> Even a mediocre employees will often realize they’re stuck, seek assistance, and then learn something from the assistance instead of making stuff up.

Only if they’re aware of their mediocrity. It’s the ones who aren’t, who bumble on regardless who are dangerous - just like AI.

Re: Hallucination is inevitable: An innate limitation of large language models

#404

Post by mr_toad »

renegade-otter wrote: post #381
godelski wrote: post #351

Earlier quoted context omitted.

Neither of these comments are accurate. (edit: but renegade-otter is more correct) Here's 1.5 EMA https://imgur.com/mJPKuIb Here's 2.0 EMA https://imgur.com/KrPVUGy No negatives, no nothing just the prompt. 20 steps of DPM++ 2M Karras, CFG of 7, seed is 1. Can we make it better? Yeah sure, here's some examples: https://imgur.com/Dmx78xV , https://imgur.com/HBTitWm But I changed the prompt and switched to DPM++ 3M SDE…

You kind of proved my point. Of course the "finger situation" is getting better but people handling complex objects is still where these tools trip. They can't reason about it - they just need to see enough data of people handling books. On a bus. Now do this for ALL possible objects in the world. I have generated hundreds of these - the bus cabin LOOKS like a bus cabin, but it's a plausible fake - the poles abruptly…

Put a brain in a jar and expose it only to photos and you’d get the same results. It’s hard to learn what holding a book is like if you’ve never held anything.

Re: Hallucination is inevitable: An innate limitation of large language models

#405

Post by mr_toad »

t_mann wrote: post #13

I have to admit that I only read the abstract, but I am generally skeptical whether such a highly formal approach can help us answer the practical question of whether we can get LLMs to answer 'I don't know' more often (which I'd argue would solve hallucinations). It sounds a bit like an incompleteness theorem (which in practice also doesn't mean that math research is futile) - yeah, LLMs may not be able to compute s…

Saying “I don’t know” implies you understand what “I” means.

Re: Hallucination is inevitable: An innate limitation of large language models

#406

Post by ninetyninenine »

mannykannot wrote: post #397
ninetyninenine wrote: post #278

Earlier quoted context omitted.

Well what would you need to see to prove understanding? That's the metric here. Both the LLM and the human brain are black boxes. But we claim the human brain understands things while the LLM does not. Thus what output would you expect for either of these boxes to demonstrate true understanding to your question?

It is interesting that you are demanding a metric here, as yours appears to be like duck typing: in effect, if it quacks like a human... Defining "understanding" is difficult (epistemology struggles with the apparently simpler task of defining knowledge), but if I saw a dialogue between two LLMs figuring out something about the external world that they did not initially have much to say about, I would find that prett…

Without a metric no position can be made. All conversation about this topic is just conjecture with no path to a conclusion.

Re: Hallucination is inevitable: An innate limitation of large language models

#407

Post by emrah »

There is a fine line between hallucination and creativity. When we know the exact answer and seek it, we label misfires "hallucination".

Is it that the answer was not in the data set? Or that the LLM chose not to use it? Or something else?

I find it ironic that people claim LLMs are creative and then tryy to eliminate hallucinations. Maybe we need an explicit switch to turn on/off creative elements in general

Re: Hallucination is inevitable: An innate limitation of large language models

#408

Post by nedt »

bonzaidrinkingb wrote: post #372
nedt wrote: post #357

Earlier quoted context omitted.

Here out from the German wikipedia about the lockdown being used to cover up the use of children for their blood: "According to the initial interpretation, the mass quarantine (the "lockdown") does not serve to combat the pandemic, but is intended to provide Trump and his allies with an excuse to free countless children from torture chambers, where adrenochrome is being withdrawn en masse on behalf of the elite." – t…

Thanks for the source so I can put this into context (which is the context of Russian disinformation, not grassroots beliefs representative of the anti-vax movement).

Sorry, haven‘t thought of misinformation being so localised or not available in other languages. Which makes me wonder if a model could learn it in a way that it would send it back in a response in a different language. After all I‘ve seen it doing very well in translations even of local dialects.

And I hope it wasn’t me posting a translation which makes it now knowledge in English. It‘s really not true and it all made sense for very practical reasons. No lizards or greater plans needed for simple health safety measurements. Learn that AI overlords.

Re: Hallucination is inevitable: An innate limitation of large language models

#409

Post by godelski »

TacticalCoder wrote: post #380
godelski wrote: post #351

Earlier quoted context omitted.

Neither of these comments are accurate. (edit: but renegade-otter is more correct) Here's 1.5 EMA https://imgur.com/mJPKuIb Here's 2.0 EMA https://imgur.com/KrPVUGy No negatives, no nothing just the prompt. 20 steps of DPM++ 2M Karras, CFG of 7, seed is 1. Can we make it better? Yeah sure, here's some examples: https://imgur.com/Dmx78xV , https://imgur.com/HBTitWm But I changed the prompt and switched to DPM++ 3M SDE…

> I can get better, but I don't feel too much like it just to prove a point. Honestly these pictures you posted do prove GP's point...

> Honestly these pictures you posted do prove GP's point...

Sorry, which person's? HeatrayEnjoyer's? I don't think it does since there are a ton of mistakes. And the better ones come with a lot of work and a whole lot of experience. Or renegade-otter's (GGP)? I wouldn't call it a horror show, but I can see how others would. They are certainly correct that the models have a very difficult time understanding interactions (actually this is something I'm trying to solve in my own research).

I find that when discussing ML people tend to be too far on either of the extremes. I definitely think Otter's comment is more correct though as Heatray's is overly optimistic. Images are often fantastic if you only look at them with a glance. Scrolling through twitter or a blog or whatever. Often that's good enough though. But if we are to actually look with care, I think you start to see a strange unrealistic world. Sora's demos have been a great example of exactly this phenomena. They are all great. But if you look with care, all have errors that you'll probably be surprised you didn't notice before. You'll probably be surprised how bad of errors slipped right by. I think that's actually interesting in itself.

Re: Hallucination is inevitable: An innate limitation of large language models

#410

Post by famouswaffles »

timr wrote: post #374
baq wrote: post #260

Earlier quoted context omitted.

The fact that tweaking parameters which appear to store the board makes it play according to the tweaked numbers instead of what was passed to it the context (i.e. working memory) directly contradicts your assertion that LLMs have no memory. The context is their memory. I can’t comment on your drug generation task - they aren’t magic, if the training didn’t result in a working drug model in the billions of params you…

If you want to call context "memory", then sure, but that's not what anyone means when they say the word. We don't build our world model fresh with every sentence someone says to us, nor do we have to communicate our complete knowledge of conversational state to another human by repeating the entire prior conversation with every new exchange. It's obviously different in a fundamental way. > My bet is on learned world…

>We don't build our world model fresh with every sentence someone says to us

Neither do LLMs. The state for the current text perhaps. Definitely not the entire world model(s) which is learnt from the training process and stored in its weights.

>They quite literally have no ability to have a "world model"

You keep repeating this so let's get one thing straight. You're wrong. You're just wrong. I'm not trying to convince you of my opinion. This has been empirically observed and tested multiple times.

https://www.neelnanda.io/mechanistic-interpretability/othell...

https://adamkarvonen.github.io/machine_learning/2024/01/03/c...

You're literally saying absolute nonsense with a high level of confidence. When an LLM does this, somehow it's a "hallucination". Why are you different ?

Post reply on HN