Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Researchers describe how to tell if ChatGPT is confabulating

arstechnica.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Researchers describe how to tell if ChatGPT is confabulating

#21

Post by Terr_ »

derefr wrote: post #12

> But perhaps the simplest explanation is that an LLM doesn't recognize what constitutes a correct answer but is compelled to provide one Why is it compelled to provide one, anyway? Which is to say, why is the output of each model layer a raw softmax — thus discarding knowledge of the confidence each layer of the model had in its output? Why not instead have the output of each layer be e.g. softmax but rescaled by mi…

Careful, I think there's a large difference here between:

1. An LLM's mathematical "confidence" of having a clear best-scoring candidate for the predicted next token when given a list of tokens.

2. A not-yet-invented AI that models the idea of different entities interacting, the concept of questions and answers, the concept of logical conflicts, and it's "confidence" that a proposition is compatible with other "true" propositions and incompatible with false ones.

To help illustrate the difference, suppose you trained an LLM on texts where a particular question was always answered with "I don't know, I have zero confidence in anything anymore." Later the LLM will regurgitate similarly nihilistic text, and by all objective internal measures it will be extremely "confident" as it does so.

> Why is it compelled to provide one, anyway

It's following the patterns in its training data, which probably reflects a whole lot more people trying to provide answers (sometimes even deliberately wrong ones) as opposed to admitting uncertainty.

This is especially true if developers put their thumb on the scale by injecting primer-text like "You are an intelligent computer eager to provide answers", as opposed to "behave like Socrates and help people understand that nothing is truly knowable."

Re: Researchers describe how to tell if ChatGPT is confabulating

#22

Post by Terr_ »

glymor wrote: post #5

TL;DR sample the top N results from the LLM and use traditional NLP to extract factoids, if the LLM is confabulating the factoids would have random distribution, but if it's not it will be heavily weighted towards one answer. A figure from the paper shows this better than my TL;DR: https://www.nature.com/articles/s41586-024-07421-0/figures/1

The LLM is already generating factoids: Things which resemble a fact without actually being one.

(See also: Androids that resemble men but aren't, asteroids that resemble stars but aren't, meteoroids that resemble meteors but aren't...)

Re: Researchers describe how to tell if ChatGPT is confabulating

#23

Post by ImHereToVote »

derefr wrote: post #12

> But perhaps the simplest explanation is that an LLM doesn't recognize what constitutes a correct answer but is compelled to provide one Why is it compelled to provide one, anyway? Which is to say, why is the output of each model layer a raw softmax — thus discarding knowledge of the confidence each layer of the model had in its output? Why not instead have the output of each layer be e.g. softmax but rescaled by mi…

I don't see a lot of answers on Stack Overflow that go "gee, I don't know how to solve that." Hence the confabulation.

Re: Researchers describe how to tell if ChatGPT is confabulating

#25

Post by superb_dev »

visarga wrote: post #19
glymor wrote: post #5

TL;DR sample the top N results from the LLM and use traditional NLP to extract factoids, if the LLM is confabulating the factoids would have random distribution, but if it's not it will be heavily weighted towards one answer. A figure from the paper shows this better than my TL;DR: https://www.nature.com/articles/s41586-024-07421-0/figures/1

Or we could just ask the same question on 3 different LLMs, ideally a large LLM, a RAG LLM and a small one, then use LLM again to rewrite the final answer. When models contradict each other there is likely hallucination going on, but correct answers tend to converge.

Why use an LLM to check the work of a different LLM?

You could use the same technique that this paper describes to compare the answers each LLM gave. LLMs don’t have to be in opposition to traditional NLP techniques

Re: Researchers describe how to tell if ChatGPT is confabulating

#26

Post by mewpmewp2 »

Terr_ wrote: post #21
derefr wrote: post #12

> But perhaps the simplest explanation is that an LLM doesn't recognize what constitutes a correct answer but is compelled to provide one Why is it compelled to provide one, anyway? Which is to say, why is the output of each model layer a raw softmax — thus discarding knowledge of the confidence each layer of the model had in its output? Why not instead have the output of each layer be e.g. softmax but rescaled by mi…

Careful, I think there's a large difference here between: 1. An LLM's mathematical "confidence" of having a clear best-scoring candidate for the predicted next token when given a list of tokens. 2. A not-yet-invented AI that models the idea of different entities interacting, the concept of questions and answers, the concept of logical conflicts, and it's "confidence" that a proposition is compatible with other "true"…

Also to some questions it also is already overly cautious about answering. E.g. I give an image of a location when I am travelling and ask it to guess where the image is taken. It will not want to guess and it will at first provide a long disclaimer that it can't do it, but if I tell it, that it is a game and just make a guess for the fun of it, it is surprisingly accurate.

Re: Researchers describe how to tell if ChatGPT is confabulating

#27

Post by bubblyworld »

derefr wrote: post #12

> But perhaps the simplest explanation is that an LLM doesn't recognize what constitutes a correct answer but is compelled to provide one Why is it compelled to provide one, anyway? Which is to say, why is the output of each model layer a raw softmax — thus discarding knowledge of the confidence each layer of the model had in its output? Why not instead have the output of each layer be e.g. softmax but rescaled by mi…

Changing the internals of a net is more likely to affect its training speed and ability to converge than its observable behaviour after training (in my experience), and in general mutations make things worse unless you have good reason to believe it'll make things better (e.g. residual layers in resnets).

(disclaimer: I'm not an ML expert, maybe this is just a me problem, but I find nets extremely sensitive to stuff like changing activations, adding normalisation, layer initialisation, layer sizes, all this stuff that seems kinda arbitrary to a non-experts like me)

Re: Researchers describe how to tell if ChatGPT is confabulating

#28

Post by amenhotep »

dawatchusay wrote: post #3

Is confabulation different from hallucination? If not I do suppose this is a more accurate term for the phenomenon except that the exact definition isn’t common sense without looking it up whereas “hallucination” is more widely understood.

Some people seem to get very angry about calling it "hallucination", because it's a computer, computers can't hallucinate! Stop anthropomorphising it!!

So I suppose if you want to stay on the right side of those people - or you are one - you call it confabulation instead.

Re: Researchers describe how to tell if ChatGPT is confabulating

#29

Post by williamcotton »

amenhotep wrote: post #28
dawatchusay wrote: post #3

Is confabulation different from hallucination? If not I do suppose this is a more accurate term for the phenomenon except that the exact definition isn’t common sense without looking it up whereas “hallucination” is more widely understood.

Some people seem to get very angry about calling it "hallucination", because it's a computer, computers can't hallucinate! Stop anthropomorphising it!! So I suppose if you want to stay on the right side of those people - or you are one - you call it confabulation instead.

There’s also the position that a definition of confabulate…

To fill in gaps in one's memory with fabrications that one believes to be facts.

…is much more accurate.

Since we’re talking about a technical process it helps to be more precise in our use of language.

Re: Researchers describe how to tell if ChatGPT is confabulating

#30

Post by todd8 »

dawatchusay wrote: post #3

Is confabulation different from hallucination? If not I do suppose this is a more accurate term for the phenomenon except that the exact definition isn’t common sense without looking it up whereas “hallucination” is more widely understood.

Humans too can be committed to beliefs that are not true. I have friend that believes in and regularly consults her "clairvoyant". I wonder if our AI assistants in the future will be vulnerable to suspicions or popular fantasies about the world, people, or even other AIs they interact with.
Post reply on HN