Researchers describe how to tell if ChatGPT is confabulating
arstechnica.com
Researchers describe how to tell if ChatGPT is confabulating
1–10 of 39 posts
Re: Researchers describe how to tell if ChatGPT is confabulating
#2> LLMs aren't trained for accuracy
This assertion in the article doesn't seem right at all. When LLMs weren't trained for accuracy, we had "random story generators" like GPT-2 or GPT-3. The whole breakthrough with RLHF was that we started training them for accuracy - or the appearance of it, as rated by human reviewers.
This step both made the models a lot more useful and willing to stick to instructions, and also a lot better at... well, sounding authoritative when they shouldn't.
Re: Researchers describe how to tell if ChatGPT is confabulating
#3Is confabulation different from hallucination? If not I do suppose this is a more accurate term for the phenomenon except that the exact definition isn’t common sense without looking it up whereas “hallucination” is more widely understood.
Re: Researchers describe how to tell if ChatGPT is confabulating
#4> LLMs aren't trained for accuracy This assertion in the article doesn't seem right at all. When LLMs weren't trained for accuracy, we had "random story generators" like GPT-2 or GPT-3. The whole breakthrough with RLHF was that we started training them for accuracy - or the appearance of it, as rated by human reviewers. This step both made the models a lot more useful and willing to stick to instructions, and also a…
Isn't that the issue? Getting thumbs up from an underpaid human reviewer isn't the same as accurate facts.
Re: Researchers describe how to tell if ChatGPT is confabulating
#5TL;DR sample the top N results from the LLM and use traditional NLP to extract factoids, if the LLM is confabulating the factoids would have random distribution, but if it's not it will be heavily weighted towards one answer.
A figure from the paper shows this better than my TL;DR: https://www.nature.com/articles/s41586-024-07421-0/figures/1
Re: Researchers describe how to tell if ChatGPT is confabulating
#6> LLMs aren't trained for accuracy This assertion in the article doesn't seem right at all. When LLMs weren't trained for accuracy, we had "random story generators" like GPT-2 or GPT-3. The whole breakthrough with RLHF was that we started training them for accuracy - or the appearance of it, as rated by human reviewers. This step both made the models a lot more useful and willing to stick to instructions, and also a…
Isn't that the issue? Getting thumbs up from an underpaid human reviewer isn't the same as accurate facts.
It's not the same as completely accurate facts, but it's much closer to accurate facts than LLMs we had before.
Re: Researchers describe how to tell if ChatGPT is confabulating
#7Is confabulation different from hallucination? If not I do suppose this is a more accurate term for the phenomenon except that the exact definition isn’t common sense without looking it up whereas “hallucination” is more widely understood.
When speaking about LLMs, confabulation and hallucination refer to the same thing. The term "confabulation" is just the most accurate description of what's happening, whereas the term "hallucination" refers to something LLMs are fundamentally incapable of.
Re: Researchers describe how to tell if ChatGPT is confabulating
#8> There are a number of reasons for this. The AI could have been trained on misinformation; the answer could require some extrapolation from facts that the LLM isn't capable of; or some aspect of the LLM's training might have incentivized a falsehood
This article seems rather contrived. They present this totally broken idea of how LLMs work (that they are trained from the outset for accuracy on facts) and then proceed to present this research as it is a discovery that LLMs don't work like that.
Re: Researchers describe how to tell if ChatGPT is confabulating
#9> LLMs aren't trained for accuracy This assertion in the article doesn't seem right at all. When LLMs weren't trained for accuracy, we had "random story generators" like GPT-2 or GPT-3. The whole breakthrough with RLHF was that we started training them for accuracy - or the appearance of it, as rated by human reviewers. This step both made the models a lot more useful and willing to stick to instructions, and also a…
Isn't that the issue? Getting thumbs up from an underpaid human reviewer isn't the same as accurate facts.
The one person I know getting paid to review AI outputs gets paid anywhere from $25 / hour to $40 / hour. Not sure if that's underpaid. It may be a nice option when you can do it at any time to supplement your regular income.
Re: Researchers describe how to tell if ChatGPT is confabulating
#10Is confabulation different from hallucination? If not I do suppose this is a more accurate term for the phenomenon except that the exact definition isn’t common sense without looking it up whereas “hallucination” is more widely understood.
From the paper https://www.nature.com/articles/s41586-024-07421-0>:
> Here we develop new methods grounded in statistics, proposing entropy-based uncertainty estimators for LLMs to detect a subset of hallucinations—confabulations—which are arbitrary and incorrect generations.