Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in…
> It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something , instead of collapsing onto lack of answer tokens.
Until the big labs decide to start adding synthetic "I don't know" outcomes to their data sets, I've been thinking that the best way to evaluate uncertainty is to have a separate LLM monitoring the conversation and asking it to classify if the agent is overstating its confidence. On the other hand I've also noticed that most models will tell you they don't know something if you specifically include it in the prompt, eg "if you don't know the answer, just say so" and/or "be clear about any gaps in your knowledge that would reduce the confidence of your response" etc. but even with the big frontier models I have noticed some quality degradation if I throw too many instructions into the system prompt. I have a little more faith in harness-level engineering than in praying to the token generation gods.
That said, there is a completely different form of "uncertainty" in which the LLM tends to place very high trust in its own prior outputs as well as user provided inputs. Again if you look at the thinking traces, these models will try very very hard to rationalize the inputs they are given, falling back to the possibility of user error only after working through several alternative possibilities, maybe even investigating data or source code in the process. And if your context is big enough, the model might just completely miss when pieces of information conflict.