Reasoning models don't always say what they think
anthropic.com
Reasoning models don't always say what they think
1–10 of 279 posts
Re: Reasoning models don't always say what they think
#2Sounds like LLMs short-circuit without necessarily testing their context assumptions.
I also recognize this from whenever I ask it a question in a field I'm semi-comfortable in, I guide the question in a manner which already includes my expected answer. As I probe it, I often find then that it decided to take my implied answer as granted and decide on an explanation to it after the fact.
I think this also explains a common issue with LLMs where people get the answer they're looking for, regardless of whether it's true or there's a CoT in place.
Re: Reasoning models don't always say what they think
#3seemed common-sense obvious to me -- AI (LLMs) don't "reason". great to see it methodically probed and reported in this way.
but i am just a casual observer of all things AI. so i might be too naive in my "common sense".
Re: Reasoning models don't always say what they think
#4I recently had fascinating example of that where Sonnet 3.7 had to decide for one option from a set of choices.
In the thinking process it narrowed it down to 2 and finally in the last thinking section it decided for one, saying it's best choice.
However, in the final output (outside of thinking) it then answered with the other option with no clear reason given
Re: Reasoning models don't always say what they think
#5Sounds like LLMs short-circuit without necessarily testing their context assumptions. I also recognize this from whenever I ask it a question in a field I'm semi-comfortable in, I guide the question in a manner which already includes my expected answer. As I probe it, I often find then that it decided to take my implied answer as granted and decide on an explanation to it after the fact. I think this also explains a…
i found with the gemini answer box on google, it's quite easy to get the answer you expect. i find myself just playing with it, asking a question in the positive sense then the negative sense, to get the 2 different "confirmations" from gemini. also it's easily fooled by changing the magnitude of a numerical aspect of a question, like "are thousands of people ..." then "are millions of people ...". and then you have the now infamous black/white people phrasing of a question.
i haven't found perplexity to be so easily nudged.
Re: Reasoning models don't always say what they think
#6Sounds like LLMs short-circuit without necessarily testing their context assumptions. I also recognize this from whenever I ask it a question in a field I'm semi-comfortable in, I guide the question in a manner which already includes my expected answer. As I probe it, I often find then that it decided to take my implied answer as granted and decide on an explanation to it after the fact. I think this also explains a…
This is such an annoying issue in assisted programming as well.
Say you’re referencing a specification, and you allude to two or three specific values from that specification, you mention needing a comprehensive list and the LLM has been trained on it.
I’ll often find that all popular models will only use the examples I’ve mentioned and will fail to elaborate even a few more.
You might as well read specifications yourself.
It’s a critical feature of these models that could be an easy win. It’s autocomplete! It’s simple. And they fail to do it every single time I’ve tried a similar abstract.
I laugh any time people talk about these models actually replacing people.
They fail at reading prompts at a grade school reading level.
Re: Reasoning models don't always say what they think
#7Sounds like LLMs short-circuit without necessarily testing their context assumptions. I also recognize this from whenever I ask it a question in a field I'm semi-comfortable in, I guide the question in a manner which already includes my expected answer. As I probe it, I often find then that it decided to take my implied answer as granted and decide on an explanation to it after the fact. I think this also explains a…
The LLMs copy human written text, so maybe they'll implement Motivated Reasoning just like humans do?
Or maybe it's telling people what they want to hear, just like humans do
Re: Reasoning models don't always say what they think
#8One interesting quirk with Claude is that it has no idea its Chain-of-Thought is visible to users.
In one chat, it repeatedly accused me of lying about that.
It only conceded after I had it think of a number between one and a million, and successfully 'guessed' it.
Re: Reasoning models don't always say what they think
#9The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it "explain itself"?
CoT improves results, sure. And part of that is probably because you are telling the LLM to add more things to the context window, which increases the potential of resolving some syllogism in the training data: One inference cycle tells you that "man" has something to do with "mortal" and "Socrates" has something to do with "man", but two cycles will spit those both into the context window and lets you get statistically closer to "Socrates" having something to do with "mortal". But given that the training/RLHF for CoT revolves around generating long chains of human-readable "steps", it can't really be explanatory for a process which is essentially statistical.
Re: Reasoning models don't always say what they think
#10Sounds like LLMs short-circuit without necessarily testing their context assumptions. I also recognize this from whenever I ask it a question in a field I'm semi-comfortable in, I guide the question in a manner which already includes my expected answer. As I probe it, I often find then that it decided to take my implied answer as granted and decide on an explanation to it after the fact. I think this also explains a…
The LLMs copy human written text, so maybe they'll implement Motivated Reasoning just like humans do? Or maybe it's telling people what they want to hear, just like humans do
They definitely tell people what they want to hear. Even when we'd rather they be correct, they get upvoted or downvoted by users, so this isn't avoidable (but is is fawning or sychophancy?)
I wonder how deep or shallow the mimicry of human output is — enough to be interesting, but definitely not quite like us.