Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

251–260 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#251

Post by yreg »

epihelix wrote: post #238
yreg wrote: post #86

Earlier quoted context omitted.

The null hypothesis is that we don't know jack shit about consciousness. Any claim of certainty seems extraordinary to me and I want to hear the evidence.

That's not a null hypothesis, that's a strawman. Regardless -- if we don't know jack shit about consciousness (your words), then any claims of LLMs being conscious are by definition untestable, pure speculation and based on no evidence at all.

> then any claims of LLMs being conscious are by definition untestable, pure speculation and based on no evidence at all.

Am I saying otherwise?

What's the correct null hypothesis according to you?

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#252

Post by user43928 »

wizzwizz4 wrote: post #250
user43928 wrote: post #248

Earlier quoted context omitted.

What is put in the cache?

Things that the software running the model would otherwise recompute, if not for the cache. What special meaning are you assigning to it?

Beats me how it works, honestly can't wrap my head around it.

From what I understand, at position Aha in each layer it's constructing a query based on the current activation and looking at the key of each other token position for that layer, in order to decide how much attention to pay to the value.

In this way it attends to the previous values, such as perhaps the incorrect assumption and plausible explanation.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#253

Post by JohnMakin »

phoghed wrote: post #190
JohnMakin wrote: post #35

Earlier quoted context omitted.

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…

> I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. Unless doing so changed the compiler output, which is what happens when you say different things to an LLM.

code changes change the compiler output.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#254

Post by altmanaltman »

tavavex wrote: post #241
altmanaltman wrote: post #138

Earlier quoted context omitted.

> Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were. I mean, that is the entire definition of the word. And you also anthropomorphize living beings like many people genuinely attach human qualities to their pets etc. Yes, the risks are very high when it comes to chatbots in particular, especially to people who are not technically inclined. But you'll be surpr…

> I mean, that is the entire definition of the word. Not quite - my wording there was very deliberate. By saying that it's a problem when you're treating something that's not living (not non-human!) as if it were, that excludes pets and all animals from the equation. I understand how common it is for humans to assign human qualities to other things and beings, but there is also an unspoken variable of intensity. Repr…

No its not unspoken. Read the paper linked. It explains what you think you are explaining but with scientific rigor.

And you are wrong in your defintion of the world. Attaching human qualities to any non human entity (living or otherwise) is the accepted defintion of what anthropomorphizing is. It does not only apply to non-living entities.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#255

Post by nullc »

wizzwizz4 wrote: post #9
Jeff_Brown wrote: post #5

Earlier quoted context omitted.

By itself, "aha" carries no insight, but the insight is probably stated immediately after it. In that case the aha is semantically useful, by identifying the insight it is near.

> but the insight is probably stated immediately after it. If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state. There is no reason to draw the conclusion you've drawn. Furthermore, what LLMs are doing isn't thought.

> If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state.

Yes they do, they have their KV caches-- it's a pure function of the input tokens, sure but that doesn't prevent it from containing latent 'insight'. LLMs can and do pre-form the tokens they're expecting to output multiple steps in the future.

I wouldn't argue that the 'aha' means anything, but the structural argument that it can't that I think you're making isn't sound.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#256

Post by nullc »

basedpolymer wrote: post #8

The anthropomorphization of LLMs should be discouraged as much as possible. It perpetuates bad practices and encourages the use of these bots for tasks they are not intended for (particularly as chatbots). Thinking traces should be treated as black boxes. There is no point in reading them. Only the LLMs’ conclusions are relevant. This is particularly true of Opus 5, which employs reasoning that seems highly questiona…

Anthropic's reasoning output isn't the real model reasoning but some sloppified summary of it.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#257

Post by cheesecakegood »

kelnos wrote: post #147
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

> Is anthropomorphizing a real problem? Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness. Your bit with MySQL is harmless because it's obvious that a database isn't a sentient lifeform. But LLMs can look like they're the real deal, and people believe it is. Using terminology like "thinking" and "reasoning" to describe what they do only reinforces this. Hav…

I think it was Karpathy that suggested he preferred “simulated reasoning” but sadly it didn’t catch on.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#258

Post by throw310822 »

wizzwizz4 wrote: post #9
Jeff_Brown wrote: post #5

Earlier quoted context omitted.

By itself, "aha" carries no insight, but the insight is probably stated immediately after it. In that case the aha is semantically useful, by identifying the insight it is near.

> but the insight is probably stated immediately after it. If the intermediate tokens represent reasoning or thought, you would expect "aha" to occur after the thoughts that led to the realisation, including the thoughts encoding the explanation: they don't have any other state. There is no reason to draw the conclusion you've drawn. Furthermore, what LLMs are doing isn't thought.

Consider this: while the inner state of an LLM (all its activations, residuals stream that is cached in the KV cache) is fully deterministic given its input sequence, the information contained in it IS NOT identical to the information in the input sequence. The reason is obvious: the LLM itself contains an enormous amount of information in its parameters and it transfers it to its residuals stream at each forward pass.

In other words: the final state given the two input sequences (where NT stands for "null token"):

[NT]

and

[NT] [NT] [NT] [NT] [NT] [NT] [NT] [NT]

is not the same, and at each forward pass the LLM keeps working on the solution even if the input tokens provide absolutely no further information.

If this is correct, then there is no need for the model to have already verbalized the key elements that drive the "aha" moment, so no need for the "aha" to appear after a full explanation.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#259

Post by zeryx »

ModernMech wrote: post #233
wrsh07 wrote: post #231

Earlier quoted context omitted.

To quote Sarah Constantin: > Humans Who Are Not Concentrating Are Not General Intelligences^ But yeah, you should still treat them with humanity. (Related: you should treat LLMs well, not because they're human but because you are^^) ^ https://srconstantin.github.io/2019/02/25/humans-who-are-not... ^^ hmm couldn't find this tweet but didn't look too hard

I gotta say though, turning on caps lock does seem to get them to cut out the inane banter.

My favorite - reply like I do, review my previous messages to understand my tone and verbosity. As much as needed, as little as possible

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#260

Post by ModernMech »

zeryx wrote: post #259
ModernMech wrote: post #233

Earlier quoted context omitted.

I gotta say though, turning on caps lock does seem to get them to cut out the inane banter.

My favorite - reply like I do, review my previous messages to understand my tone and verbosity. As much as needed, as little as possible

It'll end up like that episode of Dr. Who where they get stranded in a lunar tour bus -- slowly starting to mimic you, and then it'll predict and replace you.
Post reply on HN