Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hallucination is inevitable: An innate limitation of large language models

arxiv.org

Post reply on HN
Print view View original

51–60 of 491 posts

Re: Hallucination is inevitable: An innate limitation of large language models

#51

Post by timr »

baq wrote: post #4
paulsutter wrote: post #3

The models are just generating probable text. What’s amazing of how often the text is correct. It’s no surprise at all when it’s wrong Their bold confidence to be flat out wrong may be their most human trait

This again. They’re trained to generate probable text. The mechanisms created in the parameter blob during training to do that are basically a mystery and have to be pulled out of the model with digital brain surgery. E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those paramete…

> E.g. LLMs are reasonable at chess and turns out somewhere in the blob there’s a chessboard representation, and you can make the model believe the board is in a different state by tweaking those parameters.

Broadly agreed, but there's no "representation"...the model has no memory, let alone a "concept" of a chessboard. It's just trained on a bunch of textual replays of chess games, and this works well enough for a game with finite probability space.

Likewise, I have asked generic LLMs to generate "novel" drugs that solve particular problems, because their training sets have included lots of examples of drug molecules in string form (aka SMILES strings). This works far less well (because chemical space is much larger than a chess game), and the "novel" drugs usually end up looking like some mishmash of existing chemicals for the same problem. This can be a useful tool, but nobody is confusing it for a human brain doing problem solving.

People are easily misled by the purported intelligence of these things because they assume that common examples of human intelligence are less probable than they really are. Our languages and games and stories are pretty regular, all things considered. Expand the probability space to something truly vast (even images work for this), and you can easily see the stochastic parrot emerge.

Re: Hallucination is inevitable: An innate limitation of large language models

#52

Post by tgv »

gardenhedge wrote: post #38
tgv wrote: post #32

Earlier quoted context omitted.

Asking it to write code for you is basically asking it to hallucinate.

I don't think so. I think it's asking it to repeat code it has been trained on

Search for a piece of code you wrote. If it's more than 15 lines and not boilerplate, chances are you won't find it anywhere on the net.

Re: Hallucination is inevitable: An innate limitation of large language models

#53

Post by hyperpape »

The result seems to rely on stipulating the LLM must answer true or false to all its questions, and can't say "I don't know." So it's an interesting result, but it's not obvious that it tells us much about our actual problem, which is 100% about how to get a system that accurately understand the level of confidence it should have.

Re: Hallucination is inevitable: An innate limitation of large language models

#54

Post by itsacomment »

avidphantasm wrote: post #42
avidphantasm wrote: post #24

Earlier quoted context omitted.

Perhaps solving hallucinations at the LLM level alone is impossible, hence the inevitability. I reckon that lots of human “hallucination” is simply caught by higher-level control loops operating over the output of the generative mechanism. Basically, our conscious mind says, “nah, that doesn’t look right” enough that most of the time most of us don’t “hallucinate”.

So this implies that instead of spending resources on training bigger and bigger LLMs, AI practitioners need to shift focus to developing “ontological” and “epistemological” control loops to run on top of the LLM. I suspect they already have rudimentary such control loops. In a sense, the “easier” part of AI may be a largely “solved” problem, leaving the development of “consciousness” to be solved, which is obviously…

Our brains are very modular. I'd not be surprised at all if a similarly modular structure would turn out to be the next big step for LLMs.

Re: Hallucination is inevitable: An innate limitation of large language models

#55

Post by cubefox »

timini wrote: post #45
az09mugen wrote: post #37

Earlier quoted context omitted.

I think there is no easy way to make an LLM answer "I don't know". For that, it should learn among all the stuff ingested when people effectively don't know. But most people on internet write down irrelevant stuff even when they don't know instead of simply writing "I don't know". That's a very good point.

I think its fairly simple, it needs a certain level of proof e.g references to authoritative sources, if not say "i don't know".

Then it is nothing more than a summarizer for search engine results.

Re: Hallucination is inevitable: An innate limitation of large language models

#56

Post by card_zero »

Certhas wrote: post #50
timini wrote: post #45

Earlier quoted context omitted.

I think its fairly simple, it needs a certain level of proof e.g references to authoritative sources, if not say "i don't know".

LLMs don't have a concept of sources for their statements. Ask them to give you some literature recommendations on something it has explained to you. You'll get plenty of plausible sounding papers that don't exist. Humans know to some extent why they know (read it in a text book, colleague mentioned it). LLMs don't seem to.

They read it in a non-existent average interpolation of the books actual humans read similar things in.

Re: Hallucination is inevitable: An innate limitation of large language models

#57

Post by corimaith »

avidphantasm wrote: post #42
avidphantasm wrote: post #24

Earlier quoted context omitted.

Perhaps solving hallucinations at the LLM level alone is impossible, hence the inevitability. I reckon that lots of human “hallucination” is simply caught by higher-level control loops operating over the output of the generative mechanism. Basically, our conscious mind says, “nah, that doesn’t look right” enough that most of the time most of us don’t “hallucinate”.

So this implies that instead of spending resources on training bigger and bigger LLMs, AI practitioners need to shift focus to developing “ontological” and “epistemological” control loops to run on top of the LLM. I suspect they already have rudimentary such control loops. In a sense, the “easier” part of AI may be a largely “solved” problem, leaving the development of “consciousness” to be solved, which is obviously…

When I studied NLP, Language Models were only one part of a chatbot system used to handle language input and output. The "internal" reasoning would be handled by a knowledgeable representation systems. I guess that's the closest part to a true general AI.

The first order predicate logic we studied had alot of limitations in fully expressing real knowledge, and developing better models delves deep into the foundations of logic and mathematics. I would imagine this is a problem that has less to do with funding than requiring literal geniuses to solve. And that goes back into the pitfalls of the AI winters.

Re: Hallucination is inevitable: An innate limitation of large language models

#58

Post by barrkel »

timini wrote: post #45
az09mugen wrote: post #37

Earlier quoted context omitted.

I think there is no easy way to make an LLM answer "I don't know". For that, it should learn among all the stuff ingested when people effectively don't know. But most people on internet write down irrelevant stuff even when they don't know instead of simply writing "I don't know". That's a very good point.

I think its fairly simple, it needs a certain level of proof e.g references to authoritative sources, if not say "i don't know".

LLMs are token completion engines. The correspondence of the text to the truth or authoritative sources is a function of being trained on text like that; with the additional wrinkle that generalization from training (a desired property or it's just a memorization engine) will produce text which is only plausibly truthful, it only resembles training data.

Getting beyond this is a tricky dark art. There isn't any simple there. There's nowhere to put an if statement.

Re: Hallucination is inevitable: An innate limitation of large language models

#59

Post by Seb-C »

carlossouza wrote: post #31

Someone smart once said: If it is good, we call it "creativity." If it is bad, we call it "hallucination." This isn't a bug (or limitation, as the authors say). It's a feature.

That is correct, it's always hallucinating and making things up.

Just because those hallucinations sometimes randomly happens to be right, people concluded that being wrong is the exception, while being right is somehow the rule.

It's like when people read [insert millenias old text here], finds a part that happens to illustrate something in their life today and conclude that it is a prophecy that predicted the future.

The meaning/truth in those is nothing more than a cognitive bias from the mind of the reader, not an inherent quality of the text.

Re: Hallucination is inevitable: An innate limitation of large language models

#60

Post by cjdell »

I thought the industry was already experimenting with the idea that you have another LLM observing the output of the primary LLM which is trained more towards safety than creativity.

On top of that it would be good if the safety LLM could give a confidence score in the answer given by the main LLM. Then you can try multiple attempts with different parameters and only show the highest confidence answer to the user.

Post reply on HN