Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

61–70 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#61

Post by DoctorOetker »

Peculiarly vocal, where were all these people when they started calling the machines computers, anthropomorphizing them akin to the original human (most often female) computers that used to run such calculations? And how dangerous the consequences, we've been dead reckoning for 60-70 years with the wrong terminology without course correction!

Where were these vocal people when the "raster-oriented ink deposition machines" were being called "printers"? The meat or machine brains of future historians will melt because they can't handle ambiguity, a word gaining extra -yet similar- meaning! A word with multiple meanings, unheard of!

Where were these vocal people when people started using software terminology like "executing", "calling", "throwing and catching errors", as if software were human -clownlike sure- but human?

The danger!

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#62

Post by astrange »

Groxx wrote: post #44
c0_0p_ wrote: post #13

Earlier quoted context omitted.

I don't think there's anything like that going on. They just word vomit into a secondary area, and then there is an internal prompt that says "clean this up and summarize for the user".

Less "internal prompt" and more "they are trained to summarize after a token"

The training methods try not to apply any particular rules to the contents of the thinking text. That's called "optimization pressure on CoT" and is thought to reduce safety by inducing the model to lie (or stop clearly printing its intentions) in the thinking text.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#63

Post by pera »

kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

Yes it is a very serious problem because it confuses a lot of folks with a great deal of power like judges and policymakers.

The first book I ever read on ML (late 90s) dedicated the entire first or second chapter exploring the distinctions between artificial and biological neurons, and even talked a bit about the philosophy of modelling. I still remember thinking back then why would the authors spend so many pages on this but now I believe it was because they understood that a metaphor can be a double-edged sword.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#64

Post by hellohello2 »

mpalmer wrote: post #47
hellohello2 wrote: post #36

Earlier quoted context omitted.

Sure, and the parent comment's position is that they dislike it. Its purpose is to advocate against clickbait titles becoming normalized in the scientific community.

This is the opposite of clickbait. The topic is obvious from the title.

Clickbait doesn't have to be false, it has to be shocking. Being false is one way of being shocking. "Stop doing X!" -- really now?

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#65

Post by fedpost »

qarl2 wrote: post #34
fedpost wrote: post #20

Earlier quoted context omitted.

I think you're ending that train of thought too early. Why does this occur? Well... We can hypothesize that these things are largely trained on internet dialogue so there's probably some correlation between threads where people are not flaming each other and the quality of the replies. They're just statistical engines so anything you can do to raise the odds of a helpful next token... I'm essentially just making shit…

Did I say "it's human and we should treat it so"? Sheesh. Yes, I agree with you entirely. I'm merely pointing out that ignoring this behavior is dumb, too. And probably not rationally based. Leads people to make crazy jumps. :)

Yeah sorry, I read too far into your position. There's a certain faction within these AI discussions that wants to over-anthropomorphize the LLMs in kind of a borderline spiritual way.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#66

Post by eli_gottlieb »

kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

I find it annoying because when I read ML papers nowadays I have to back-translate from anthropomorphized talk into actual machine talk, then mentally compare to what I actually know about brains and cognition.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#67

Post by dataviz1000 »

Although I 100% agree that the core mechanism of GRPO is purely mechanical token-by-token probability generation, because RL only rewards exact final answers, the training forces the model to develop error-correction habits. This makes the output extremely like human thinking when solving a problem. It's like the order of the thinking tokens is what causes it to get that sweet, delicious reward, and this order seems like a reflection of the human thinking process.

I created a flame graph classification of thinking-token phrases into setup, execution, decomposition, verification, error correction, surrender, and deliberation, or classified as steps in an OODA loop, which is more of a reach. It literally has a verification step and, if it finds an error, an error-correction step.

If there is a verification sequence of tokens with an error-correction sequence of tokens during RL training, it will perform better; and if humans do these steps (did you proofread your reply to this comment? did you correct it?), they will perform better — which is why it is so easy to make the anthropomorphizing metaphor.

Nonetheless, the paper is 100% correct that these machines are not thinking like humans.

https://adamsohn.com/reasoning-grid/

https://adamsohn.com/lambda-variance/

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#68

Post by davidguetta »

Im waiting for the article called "stop desantropomorphizing llms" when everybody will finally accept they think like us, partly because maybe the intelligence is universal and partly because, well the datasets are fucking human bro

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#69

Post by pixl97 »

JohnMakin wrote: post #48
traverseda wrote: post #46

Earlier quoted context omitted.

Do you have a working definition of "conscious"?

Do you? I’m not sure what you’re getting at.

I believe you should look up the work of Cameron Berg before making statements like "an LLM is" or "an LLM isn't". Empirically defining all this stuff is very difficult, and making a definition that covers all beings that can exhibit conscious behavior is much more complex than a face value examination would reveal.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#70

Post by taurath »

kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

> Is anthropomorphizing a real problem?

Even tech companies are rolling out AI training which utterly anthropomorphizes it, and leads people to think its actually intelligence. This is part of the reason for the backlash - everyone understands it bullshit marketing the second you actually try to use it.

Post reply on HN