First author here, happy to answer questions
Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)
31–40 of 46 posts
Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)
#32Of course not. Because the article uses the words 'thought' and 'reasoning' and even 'faithful' to mean something other than their normal meanings, but then expects them to behave exactly the same. Every field has terms of art, and 'reasoning' is one for LLMs. But that doesn't mean it has the same properties as 'reasoning' in other contexts, because you're not referring to the same thing. Why doesn't my asteroid belt…
Related: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces! [1] > Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct soluti…
Also posted to HN today with a couple comments: https://news.ycombinator.com/item?id=49360140#49363374
Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)
#33Of course not. Because the article uses the words 'thought' and 'reasoning' and even 'faithful' to mean something other than their normal meanings, but then expects them to behave exactly the same. Every field has terms of art, and 'reasoning' is one for LLMs. But that doesn't mean it has the same properties as 'reasoning' in other contexts, because you're not referring to the same thing. Why doesn't my asteroid belt…
Over time, I've learned to accept that many people -- even very clever ones -- are incapable of holding a metaphor at arm's length. Once they accept the words of a metaphor as applicable at all, the metaphor collapses entirely into literalism for them. They can no longer see that the metaphor was just a tool with inherent limitatation and boundaries.
Because the field of artificial "intelligence" is constructed around the idea of applying psychological metaphors to computational systems (a very powerful idea!) it's almost a worst case scenario for these people.
Suddenly, they're reversing the metaphors and applying computational schema to psychological processes ("aren't we really just stochastic parrots ourselves?!"); or, like here, they find themselves surprised and confused when they stumble across the natural boundaries of the metaphor experimentally.
It's because they never had sight of the boundaries in the first place and maybe never can quite see them. The words only make sense to them as literal equivalence, and so their surprise when they run into stuff like this is earnest and deep.
Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)
#34[deleted]
Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)
#35Of course not. Because the article uses the words 'thought' and 'reasoning' and even 'faithful' to mean something other than their normal meanings, but then expects them to behave exactly the same. Every field has terms of art, and 'reasoning' is one for LLMs. But that doesn't mean it has the same properties as 'reasoning' in other contexts, because you're not referring to the same thing. Why doesn't my asteroid belt…
Related: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces! [1] > Our findings consistently challenge the prevailing narrative that intermediate tokens constitute a semantically meaningful reasoning process. First, we observe a pronounced lack of correlation between solution correctness and trace validity—models frequently produce invalid reasoning traces even when they arrive at correct soluti…
The author of this paper is in ASU and does a lot of excellent work in this space. People should check it out. Especially the paper titled "Beyond Semantics..." His twitter is also active _and_ high SNR.
Related: Poster side dialogue and Q&A about this work at ICML. Very good. https://news.ycombinator.com/item?id=49277303
Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)
#36This paper puts words to something I’ve noticed repeatedly with LLMs, particularly Qwen3.6. When I read its reasoning, it appears to recognise the mistake and then carry on as if it hadn’t noticed it at all. > models often determine their answers based on implicit biases tied to question templates, then construct reasoning chains to justify their predetermined conclusions > its reasoning was correct right until the f…
First author here, surprised to see the paper in HN! :)
When doing the paper we noticed that models are very good at generating post-hoc plausible CoT, which to me knowledge can happen quite often with relatively easy tasks.
You might be interested in reading this other paper that came out after ours: https://arxiv.org/abs/2507.05246
Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)
#37Earlier quoted context omitted.
The current political climate is well reasoned and intentional. It might not be yours or mine, however the system is working exactly as the ones paying for it have intended.
My brother and I have been arguing about that all of our lives. He believes everything is intentional and it's just a matter of discovering who benefits. I see chaos that nobody intends or controls. His political landscape is a tapestry of conspiracy theories and mine is a fog of war. I think his is more comforting, since it admits a possibility of a rational, predictable world.
I'd synthesize those as: "There is a lot of chaos with no central plan, but every small piece happens because someone believes they will benefit."
In other words, a lot of this depends on what scale/scope is being inspected. On the high level, the world is chaos rather than a meticulous and inscrutable plan of the Illuinati Shadow Cabal. On the low level, people do things for reasons, even if they're dumb ones.
With respect to the "current political climate", I'd like to suggest that a lot of dumb or seemingly "against their own interests" stuff is due to people prioritizing costly in-group loyalty signals. Their interest in staying good with the tribe is just higher than their interest against a dumb national policy.
Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)
#38Earlier quoted context omitted.
My brother and I have been arguing about that all of our lives. He believes everything is intentional and it's just a matter of discovering who benefits. I see chaos that nobody intends or controls. His political landscape is a tapestry of conspiracy theories and mine is a fog of war. I think his is more comforting, since it admits a possibility of a rational, predictable world.
It’s intentional in the sense that actors are acting intentionally for their own benefit (or at least what they believe is beneficial) and following incentives. Not that there is a master planner who manipulates everything
The current admin seems to have quite a few long term plans they have been working towards.
Project 2025, Maralago accords.
So far the only major policy item the Trump admin seems to have not intended was the Iran War. Israel killing the intended replacement, Iran leveraging the straight of Hormuz, and dropping three Tomahawks on an elementary school really botched that one.
Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)
#39I thought this was already widely known? From March last year: https://transformer-circuits.pub/2025/attribution-graphs/bio... There's no reason to believe the model's self-reported "thinking" bears any relation to the mechanics by which it arrived at some output.
that sounds testable - if you skip the reasoning tokens, do you get the same result? if not, then there's certainly some bearing, but not necessarily in how we read the tokens as text
It is and has been - skipping the tokens causes performance to drop. But replacing the tokens with filler causes the performance to drop, but by a lot less. You can also train models to emit broken or unrelated thinking tokens and their performance is also not much worse than the ones that are trained to output somewhat coherent thinking traces.
This points to a hypothesis that the content of the tokens is only slightly related to the mechanism by which it improves performance, and that primarily the extra tokens allow the original prompt to be processed more deeply by the model, because earlier tokens will essentially pass through the model many more times than later ones.
Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)
#40Of course not. Because the article uses the words 'thought' and 'reasoning' and even 'faithful' to mean something other than their normal meanings, but then expects them to behave exactly the same. Every field has terms of art, and 'reasoning' is one for LLMs. But that doesn't mean it has the same properties as 'reasoning' in other contexts, because you're not referring to the same thing. Why doesn't my asteroid belt…
Over time, I've learned to accept that many people -- even very clever ones -- are incapable of holding a metaphor at arm's length. Once they accept the words of a metaphor as applicable at all, the metaphor collapses entirely into literalism for them. They can no longer see that the metaphor was just a tool with inherent limitatation and boundaries. Because the field of artificial "intelligence" is constructed aroun…
I don't think the boundary between "generalization" and "metaphor" is very well defined. When you go from an exemplar of 1 to 2, you're going to find all kinds of edge cases where attributes of the thing being demonstrated that had seemed to be essential turn out to not be necessary.
I think you certainly could look at LLMs as "thinking" metaphorically, but I also don't think it is necessarily only a metaphor.