Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

151–160 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#151

Post by pera »

ux266478 wrote: post #91
pera wrote: post #63

Earlier quoted context omitted.

Yes it is a very serious problem because it confuses a lot of folks with a great deal of power like judges and policymakers. The first book I ever read on ML (late 90s) dedicated the entire first or second chapter exploring the distinctions between artificial and biological neurons, and even talked a bit about the philosophy of modelling. I still remember thinking back then why would the authors spend so many pages o…

To be fair, the ANN architecture underneath is a misleading thing to be looking at, it's not where the comparison comes from. Though I can't tell if you meant it to be relevant in that way, or just as a general example for the dangerous nature of metaphor. LLMs are expressly designed to approximate human behavior within the bounds of the written word. The anthropomorphization is no more philosophically problematic th…

I meant it in the latter way: a metaphor can be useful as a pedagogical tool to introduce new ideas, and using the source of inspiration for this idea as the metaphor itself makes perfect sense, but unfortunately our brains seem to be prone to assign other properties of the metaphor that don't actually belong to the object of study.

I imagine this happens because we tend to conflate things that are similar, or maybe because it's not entirely clear which characteristics are being mapped in the metaphor?

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#152

Post by YawningAngel »

kelnos wrote: post #147
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

> Is anthropomorphizing a real problem? Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness. Your bit with MySQL is harmless because it's obvious that a database isn't a sentient lifeform. But LLMs can look like they're the real deal, and people believe it is. Using terminology like "thinking" and "reasoning" to describe what they do only reinforces this. Hav…

I think LLMs are pretty clearly intelligent in some sense of the word, and I don't know how one could ever confidently know they aren't conscious in some sense.

That's not to say they're humanlike, just that people who think they know these ideas are ridiculous seem to be overreaching in the same way Steve Yegge seems to be overreaching

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#154

Post by networked »

kelnos wrote: post #147
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

> Is anthropomorphizing a real problem? Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness. Your bit with MySQL is harmless because it's obvious that a database isn't a sentient lifeform. But LLMs can look like they're the real deal, and people believe it is. Using terminology like "thinking" and "reasoning" to describe what they do only reinforces this. Hav…

These discussions take a lot of time to get to the crux because neither side states their assumptions. Assumptions rarely get voiced at all.

I find it useful to ask:

1. Do you believe in quantum consciousness?

2. Do you think a "brain upload", a high-accuracy digital model of an organic human brain, would think or be conscious?

3. What is your working definition of thinking? It doesn't need to be rigorous.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#155

Post by throw310822 »

randomImmigrant wrote: post #98
throw310822 wrote: post #95

Earlier quoted context omitted.

> And it should not go this way, is the OPs point Well, he's wrong. If you argue that LLMs and humans are fundamentally different because all LLMs do X and no human does it, then showing you that it's not true demolishes your argument. Doesn't prove anything positive, but it certainly proves that your argument is invalid.

Well this requires you to buy the very bullshit argument that comparisons of two physically distinct systems just because they share outputs is meaningful. If I call a lightbulb an artificial star, the onus is on me to show the behavior under the hood is star like, not just to point at the light and say “you must see it’s a a star since it’s emitting light!”.

> Well this requires you to buy the very bullshit argument

Sorry, no. The only thing it requires you to buy is basic logic. If you argue that B is true because of A, the fact that A is false invalidates your argument (I repeat: not B but your argument). There is no question about it.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#156

Post by zmgsabst »

Jeff_Brown wrote: post #5
florianherrengt wrote: post #2

> While a human may say “aha” to indicate exactly a sudden internal state change, this interpretation is unwarranted for models which do not have any such internal state, and which on the next forward pass will only differ from the pre-aha pass by the inclusion of that single token in their context. Interpreting the “aha” moment as meaningful exemplifies the long-neglected assumption about long CoT models – the false…

By itself, "aha" carries no insight, but the insight is probably stated immediately after it. In that case the aha is semantically useful, by identifying the insight it is near.

“Aha” as a single token records the LLM discovered it made a mistake and needs to pivot.

On the next forward pass: it rediscovers the mistake, its “aha” noting that, and then provides the first token of the new idea.

That “aha” contains information: the previous conclusion was somehow insufficient.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#157

Post by zmgsabst »

paxys wrote: post #89
DoctorOetker wrote: post #61

Peculiarly vocal, where were all these people when they started calling the machines computers, anthropomorphizing them akin to the original human (most often female) computers that used to run such calculations? And how dangerous the consequences, we've been dead reckoning for 60-70 years with the wrong terminology without course correction! Where were these vocal people when the "raster-oriented ink deposition mach…

Computing is a task. Printing is a task. It’s not wrong to call both a human and a machine a “computer”, because computing (applying an algorithm to an input and producing an output) is literally what they are doing. Same way you can have human and machine diggers, cleaners, calculators and lots more. Anthropomorphizing comes in when we attribute much more complex behaviors to them - chain of thought, reasoning, inte…

Reasoning is mechanical.

I think a problem is that previously thought, reasoning, and intelligence were always co-occurring, but now we have machines capable of (limited) reasoning that do not think or have intelligence.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#158

Post by Dilettante_ »

nostrademons wrote: post #53
JohnMakin wrote: post #48

Earlier quoted context omitted.

Do you? I’m not sure what you’re getting at.

Not GP, but one of the challenges with debating whether LLMs are "conscious" is that we don't even really know what it means for a human to be "conscious", or even if consciousness is experienced by other humans the same way it is for ourselves. What we do know: neurons carry electrical impulses across their synapses to trigger other neurons to fire, and more frequently used synapses are strengthened while infrequent…

>"Everybody experiences the world in a different way, and yet everybody assumes that everyone else experiences the world the same way they do."

Relevant article: "Generalizing from one example"[https://www.lesswrong.com/posts/baTWMegR42PAsH9qJ/generalizi...]

Personal note: That principle has been the bane of my autistic existence. People sometimes seem literally incapable of understanding that other people even can be different.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#159

Post by Dilettante_ »

hypfer wrote: post #141
Eisenstein wrote: post #137

Earlier quoted context omitted.

Why can't it be a being? Why is thinking about such a possibilty bad?

Sorry, but if you're genuinely asking this not for trolling reasons then you should _really_ _really_ see a medical professional (not an insult). Online comment sections are not the correct place to unpack any of this. Which is indeed terminating this comment chain, but for good (and benevolent) reason. Doing anything else other than referring to a trained professional in a controlled context would potentially just f…

Nice Unthinkable Topic you have there.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#160

Post by geraneum »

kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

> Is anthropomorphizing a real problem?

Of course it is. Anthropomorphizing is in our nature, but it doesn’t mean we have to entertain it and extend it to everything. A poet can anthropomorphize clouds beautifully and I’d enjoy his poem, but I want my pilot to not see clouds as rabbits when they decide if it’s safe to fly through them.

Post reply on HN