Simplistic version of this is just asking the question in 2 ways - ask for confirmation that the answer is no, then ask for confirmation that the answer is yes :)
If it's sure it won't confirm it both ways.
Researchers describe how to tell if ChatGPT is confabulating
11–20 of 39 posts
Re: Researchers describe how to tell if ChatGPT is confabulating
#12> But perhaps the simplest explanation is that an LLM doesn't recognize what constitutes a correct answer but is compelled to provide one
Why is it compelled to provide one, anyway?
Which is to say, why is the output of each model layer a raw softmax — thus discarding knowledge of the confidence each layer of the model had in its output?
Why not instead have the output of each layer be e.g. softmax but rescaled by min(max(pre-softmax vector), 1.0)? Such that layers that would output higher than 1.0 just get softmax'ed normally; but layers that would output all "low-confidence" results (a vector all lower than 1.0) preserve the low-confidence in the output — allowing later decoder layers to use that info to build I-refuse-to-answer-because-I-don't-know text?
Re: Researchers describe how to tell if ChatGPT is confabulating
#13Simplistic version of this is just asking the question in 2 ways - ask for confirmation that the answer is no, then ask for confirmation that the answer is yes :) If it's sure it won't confirm it both ways.
A corollary for natural intelligence: if you can prove that all foos are bar, and that all foos are not bar, that's a good time to suspect that no foos actually exist.
(I still don't understand why everyone seems happy to conflate "intelligence" with fact-retrieval?)
Re: Researchers describe how to tell if ChatGPT is confabulating
#14Earlier quoted context omitted.
Isn't that the issue? Getting thumbs up from an underpaid human reviewer isn't the same as accurate facts.
The one person I know getting paid to review AI outputs gets paid anywhere from $25 / hour to $40 / hour. Not sure if that's underpaid. It may be a nice option when you can do it at any time to supplement your regular income.
Reviewing AI output or helping in training a LLM itself?
Re: Researchers describe how to tell if ChatGPT is confabulating
#15Simplistic version of this is just asking the question in 2 ways - ask for confirmation that the answer is no, then ask for confirmation that the answer is yes :) If it's sure it won't confirm it both ways.
That works for yes/no questions, but not for informational questions, like "Where's the Eiffel Tower?".
Re: Researchers describe how to tell if ChatGPT is confabulating
#16Simplistic version of this is just asking the question in 2 ways - ask for confirmation that the answer is no, then ask for confirmation that the answer is yes :) If it's sure it won't confirm it both ways.
A corollary for natural intelligence: if you can prove that all foos are bar, and that all foos are not bar, that's a good time to suspect that no foos actually exist. (I still don't understand why everyone seems happy to conflate "intelligence" with fact-retrieval?)
Because there is no single widely accepted definition of intelligence.
Re: Researchers describe how to tell if ChatGPT is confabulating
#17Earlier quoted context omitted.
A corollary for natural intelligence: if you can prove that all foos are bar, and that all foos are not bar, that's a good time to suspect that no foos actually exist. (I still don't understand why everyone seems happy to conflate "intelligence" with fact-retrieval?)
Because there is no single widely accepted definition of intelligence.
No widely accepted definition, but fact retrieval is outside of any of the ones with which I'm familiar.
Re: Researchers describe how to tell if ChatGPT is confabulating
#18What we lack is for these models to state their context for their response.
We have focused on the inherent lack of input context, leading to wrong conclusions, but what about that 90B+ parameters universe, plenty of room for multiple contexts to associate any input to surprising pathways.
In the olden days of MLPs we had the same problem with softmax basically squeezing N output scores into a normalized “probability”, where each output neuron actually was the sum of multiple weighted paths, which one winning the softmax made up the “true” answer, but there may as well have been two equally likely outcomes, with just the internal “context” as difference. In physics we have the path integral interpretation and I dare say, we humans too, may provide outputs that are shaped by our inner context.
Re: Researchers describe how to tell if ChatGPT is confabulating
#19TL;DR sample the top N results from the LLM and use traditional NLP to extract factoids, if the LLM is confabulating the factoids would have random distribution, but if it's not it will be heavily weighted towards one answer. A figure from the paper shows this better than my TL;DR: https://www.nature.com/articles/s41586-024-07421-0/figures/1
Or we could just ask the same question on 3 different LLMs, ideally a large LLM, a RAG LLM and a small one, then use LLM again to rewrite the final answer. When models contradict each other there is likely hallucination going on, but correct answers tend to converge.
Re: Researchers describe how to tell if ChatGPT is confabulating
#20Simplistic version of this is just asking the question in 2 ways - ask for confirmation that the answer is no, then ask for confirmation that the answer is yes :) If it's sure it won't confirm it both ways.
A corollary for natural intelligence: if you can prove that all foos are bar, and that all foos are not bar, that's a good time to suspect that no foos actually exist. (I still don't understand why everyone seems happy to conflate "intelligence" with fact-retrieval?)
> I still don't understand why everyone seems happy to conflate "intelligence" with fact-retrieval?
Because it's useful and impossible till very recently.