Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

21–30 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#21

Post by deaton »

Jeff_Brown wrote: post #5
florianherrengt wrote: post #2

> While a human may say “aha” to indicate exactly a sudden internal state change, this interpretation is unwarranted for models which do not have any such internal state, and which on the next forward pass will only differ from the pre-aha pass by the inclusion of that single token in their context. Interpreting the “aha” moment as meaningful exemplifies the long-neglected assumption about long CoT models – the false…

By itself, "aha" carries no insight, but the insight is probably stated immediately after it. In that case the aha is semantically useful, by identifying the insight it is near.

It really isn't useful though, unless it is a summary. At best it is a semantic trick to tell the next iteration to come up with something smart.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#22

Post by ghostpepper »

paimapi wrote: post #7
Jeff_Brown wrote: post #5

Earlier quoted context omitted.

By itself, "aha" carries no insight, but the insight is probably stated immediately after it. In that case the aha is semantically useful, by identifying the insight it is near.

it's a rhetorical heuristic that a writer should know to use when directing a reader to a declarative that they want them to pay attention to, usually because it's a non-obvious or roundabout insight when utilized by AI, it's a probabilistic output and it's variable whether or not that rhetorical trick is useful. it also pushes a non-skeptical reader to focus too much on the following text or even to believe that the…

Did not read the paper so apologies if this is covered but isn't it possible that there is some recognizable semantic pattern in the training data where an "aha" is often followed by a subtle semantic shift that proves closer to the original premise in some critical way, and by emitting the "aha" token the model causes itself to produce such a subtle semantic shift that pushes the subsequent reasoning closer to the desired response?

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#23

Post by fabsalvadori »

There is a useful engineering consequence here beyond terminology.

If intermediate tokens are not a faithful representation of the computation, then they are a pretty bad audit artifact too. We probably shouldn't be trying to make the model's internal narration more interpretable., but rather the computation around it more reproducible.

Record the actual inputs, model/version/configuration, tool observations and outputs, then make the execution replayable enough that differences between runs can be isolated.

In other words, don't ask the model to explain what it thought, and instead make the system able to show what actually happened.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#25

Post by jerf »

I'm not sure how to test this but I think there's an interesting possibility where the "reasoning" tokens are actually both an accurate reflection of a line of reasoning, but also, that there can be changes in the weights as the computation proceeds onward that may not be reflected in the apparently nominal meaning of the human language the tokens are output as for our consumption.

Some modest evidence is my own subjective experience of the many times I've explained why I'm doing something, and it is a true explanation in the sense that it is certainly not a lie, but it is also incomplete and there are entire strands of thought that went into my decision that are not being articulated. Though human speech is not equivalent to an LLM's output since we can trivially think without literally speaking whereas they can not. (No need to nitpick on the definitions there; all I'm observing here is that they are forced to emit an externally-visible artifact whereas I can sit in silence, thinking, with no externally-visible artifact being produced. Not trying to make any grand claims about what is "real" cognition or anything.)

It is conceivable how to create a test of whether the tokens correspond to the "real" thought process, and papers and work on that have been done, such as [1]. It is difficult for me to imagine how to scramble the nominal tokens without also completely trashing any implicit calculations that may be occurring too.

[1]: https://transformer-circuits.pub/2025/attribution-graphs/bio...

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#26

Post by newswasboring »

smugtrain wrote: post #14

Strong dislike for papers that tell me what to do in the title, especially when even the paper admits a loose correlation of the intermediate tokens compared to solution correctness. My solutions work and they speak for themselves.

> My solutions work and they speak for themselves.

I understand the sentiment, and I also use the "thinking" traces as insight, but wouldn't you want your solutions to be based upon a good understanding? If the correlation is weak, then our solution is also weak.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#27

Post by clhodapp »

forgotTheLast wrote: post #17
clhodapp wrote: post #4

Seems like they are closer to scratch than reasoning... Generating some scratch to draw from helps make it easier to compute the real answer.

That's my personal theory too. The model is stuffing its own context with vaguely related tokens, which helps the attention heads retrieve the right tokens.

Yup. You basically just need something for probability to push off of

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#28

Post by kgeist »

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do it too. Say, when MySQL fails to start because it tries to read its config from the wrong dir, I may say "oh, this guy thinks he must read the config from ..." (having a language with grammatical genders as my native language also helps make it sound pretty natural). It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#29

Post by jurgenburgen »

kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

> It's more fun like that :) Doesn't mean I genuinely believe a MySQL instance actually thinks.

A lot of people are not in on the joke. ELIZA effect and AI psychosis is a thing.

Interacting a lot with LLMs might be damaging to the human psyche even for mentally stable people.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#30

Post by Pannoniae »

kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

What's wrong with treating it as biology though? Even large software systems have biological aspects, their behaviour is emergent and if you want to observe how they work, a holistic approach is needed, you can't really reason about their full state...

For example, if you have a search engine or a complex game, you can't run tests like "for all inputs the results are correct", you're going to be fudging a lot, using randomness, using heuristics, and all that kinda stuff

Just like how mathematics > physics > chemistry > biology > psychology > economics/sociology (Auguste Comte's hierarchy reordered a bit for the modern day), moving up the abstraction ladder makes things more complex, less legible and less exact.

Post reply on HN