Live data from Hacker News

Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)

arxiv.org

11–20 of 46 posts

Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)

#11
I thought this was already widely known?

From March last year: https://transformer-circuits.pub/2025/attribution-graphs/bio...

There's no reason to believe the model's self-reported "thinking" bears any relation to the mechanics by which it arrived at some output.

Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)

#12
post #8

Earlier quoted context omitted.

Natural intelligences do this too

Must we always see this restated every time? It's getting a bit stale always seeing these kinds of comments on articles about LLM.

You think, "this problem" is qn LLM problem?

Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)

#13

Earlier quoted context omitted.

Natural intelligences do this too

No they don't, human intelligence has the ability to form an internal model of itself, which allows it to "notice" its own mistakes and change.

Many who watched the last decade knows just because its possible to noticed mistakes and change, its clearly not a reliable process.

Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)

#16
post #8

Earlier quoted context omitted.

Natural intelligences do this too

Must we always see this restated every time? It's getting a bit stale always seeing these kinds of comments on articles about LLM.

I can just immagine the response to a headline "LLM chooses mass death: thousands killed in horrific AI accident" being something like "lots of humans have caused mass death too."

Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)

#17

Earlier quoted context omitted.

No they don't, human intelligence has the ability to form an internal model of itself, which allows it to "notice" its own mistakes and change.

Many who watched the last decade knows just because its possible to noticed mistakes and change, its clearly not a reliable process.

The current political climate is well reasoned and intentional. It might not be yours or mine, however the system is working exactly as the ones paying for it have intended.

Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)

#18

I thought this was already widely known? From March last year: https://transformer-circuits.pub/2025/attribution-graphs/bio... There's no reason to believe the model's self-reported "thinking" bears any relation to the mechanics by which it arrived at some output.

It's... complicated. Yes, RL reward hacking makes it learn "bird language" and yes, reasoning traces can be misleading. However they also pretty clearly steer the final reply and not simply justify it, and can stay somewhat coherent and relevant with readability SFT and rewards. All these phenomenas coexist, they aren't mutually exclusive. Reasoning traces are still useful for debugging.

Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)

#19
I see this when just using some chat bot that shows the “reasoning” steps (ad-hoc observation of course, it’s really cool that people are actually studying it).

It is annoying when the bot seems be “reasoning” correctly and then makes an obvious mistake at the end. And perplexing when it seems to be completely wrong and then pull the right answer out of a magic hat at the end.

I guess it makes sense; the “reasoning” steps aren’t actually doing logic, just adding more context to influence the final generation, right? But it is weird to see.

Re: Chain-of-Thought Reasoning in the Wild Is Not Always Faithful (2025)

#20

Earlier quoted context omitted.

Many who watched the last decade knows just because its possible to noticed mistakes and change, its clearly not a reliable process.

The current political climate is well reasoned and intentional. It might not be yours or mine, however the system is working exactly as the ones paying for it have intended.

My brother and I have been arguing about that all of our lives. He believes everything is intentional and it's just a matter of discovering who benefits. I see chaos that nobody intends or controls. His political landscape is a tapestry of conspiracy theories and mine is a fog of war. I think his is more comforting, since it admits a possibility of a rational, predictable world.
Post reply on HN