Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

131–140 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#131

Post by itemize123 »

JohnMakin wrote: post #35
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…

i dont understand, isn't the arguing just some form prompt steering?

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#132

Post by a2ff6eeb0 »

knollimar wrote: post #109
red75prime wrote: post #100

Earlier quoted context omitted.

I'm not a big fan of arguments like "it's not the real [human quality], it's [mechanistic explanation]." They lack a part: "because the [human quality] allows us to do X, Y, Z, which is impossible with [this mechanism]." I agree that the relevance of retrieved pieces and the management of long-term storage could be improved, though.

It just acts fundamentally different than someone who would remember. If someone only remembered vague scraps of what you'd expect them to remember, you might say the person can't remember. It's much closer to notetaking and reviewing before responding than it is memory. The issue with anthropomorphizing like this is that "memory" comes with baggage of expectations for it to do certain things, and it breaks them. Jus…

Yeah, I think I've come to the conclusion that the biggest breakthrough we need before we can replace human thought is going to be some mechanism for live update of weights. "Learning" by injecting into context just isn't good enough.

But billions of dollars are going towards research to find these breakthroughs, so we'll get there eventually.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#133

Post by lern_too_spel »

randomImmigrant wrote: post #121
lern_too_spel wrote: post #118

Earlier quoted context omitted.

The visible light produced by both an incandescent bulb and a star is a result of black body radiation, but otherwise, I don't understand your point. A light bulb produces light, something we might have relied on stars to do before. An LLM produces thoughts, something we might have relied on people to do before. Nobody is claiming the process by which the thoughts are produced is the same, only that they both produce…

LLMs produce language. And you're right, if we restricted claims to that, no one would object. It might even be scientifically accurate, shock of shocks. If you claim LLMs produce thought, it's equivalent to claiming light bulbs undergo fission. Pure wish fulfillment. Language is not the extent of thought, and calling a language producing machine necessarily a thinking machine is an old old mistake.

> If you claim LLMs produce thought, it's equivalent to claiming light bulbs undergo fission

You're making a huge logical leap. There is nothing equivalent about these claims other than that they are made in English.

> calling a language producing machine necessarily a thinking machine is an old old mistake.

Nobody claims that all language models think. The small markov chain language models of old clearly aren't thinking and produce a lot of gibberish. The difference is that, to the surprise of many people several years ago, but to the surprise of nobody who has been following along today, the corpus of all text produced by humans contains within it information about how the world works and also information about how to reason. Using that corpus to train a sufficiently large language model causes the language model to learn a world model and a reasoning model in order to produce text that matches the training data. The reasoning model can be used to perform longer chain thinking with test time compute techniques. People who think deeply for a living recognize thinking when they see it. https://scottaaronson.blog/?p=9979

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#134

Post by FelineStateMach »

I wondered recently about why we stopped with the semantic split of thinking/actions vs user facing communication because the all powerful tool calling craze. Code comments that talk about the prompt is an obvious byproduct of the mixed context. Chain of thought and ReAct were great, but feel like a first pass moreso than the final landing spot.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#135

Post by FelineStateMach »

FelineStateMach wrote: post #134

I wondered recently about why we stopped with the semantic split of thinking/actions vs user facing communication because the all powerful tool calling craze. Code comments that talk about the prompt is an obvious byproduct of the mixed context. Chain of thought and ReAct were great, but feel like a first pass moreso than the final landing spot.

https://n.zip/2e85 I had sorta started a SFT to explore this idea. it would prove results even with smaller models that fit on a DGX spark. Just takes some more thinking rather than my ADHD mind.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#136

Post by pjerem »

JohnMakin wrote: post #35
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…

To me it’s evident that we are a few years away from the Her movie, where everyone on the street is talking to its IA friend.

I’m really afraid that it will totally destruct what is remaining of social tissue because why search for friends when you have an always on virtual (and pretty smart) friend h24 in your earbuds ?

I’m not blaming anyone for this outcome. I have myself argued with Claude more than once, and really not about code but about everyday things or nice facts of life I should rather have discussed with a friend.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#137

Post by Eisenstein »

hypfer wrote: post #127
salawat wrote: post #110

Earlier quoted context omitted.

What is terrifying is the propensity of people hoping for a mechanical slave to do everything possible to avoid touching on the possibility for being-ness of the technology they are desperately hoping will work as a basis for that implementation.

Cyberpsychosis. A next token predictor is not a being. Stop posting these things. Stop thinking these things.

Why can't it be a being? Why is thinking about such a possibilty bad?

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#138

Post by altmanaltman »

tavavex wrote: post #82
alexey-salmin wrote: post #45

Earlier quoted context omitted.

> It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently. I know people who are like that too. I'm not sure anthropomorphizing is a problem. Seeing analogies everywhere is an innate human trait, sometimes it can be harmful but more often it's useful.

Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were. Using humanizing language invites discussions of things like the rights and feelings of an algorithm. A judge that is misled by the application of human-centric language to an algorithm can lead to some terrible outcomes. Not everyone is an LLM expert and the language people use leads to them treating LLMs li…

> Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were.

I mean, that is the entire definition of the word. And you also anthropomorphize living beings like many people genuinely attach human qualities to their pets etc. Yes, the risks are very high when it comes to chatbots in particular, especially to people who are not technically inclined. But you'll be surprised at how crucial the ability of anthropomorphizing is. This is a very good paper that summarizes it and is definitely worth reading if you're interested in these things: https://www.researchgate.net/publication/5936908_On_Seeing_H...

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#139

Post by adzm »

a2ff6eeb0 wrote: post #132
knollimar wrote: post #109

Earlier quoted context omitted.

It just acts fundamentally different than someone who would remember. If someone only remembered vague scraps of what you'd expect them to remember, you might say the person can't remember. It's much closer to notetaking and reviewing before responding than it is memory. The issue with anthropomorphizing like this is that "memory" comes with baggage of expectations for it to do certain things, and it breaks them. Jus…

Yeah, I think I've come to the conclusion that the biggest breakthrough we need before we can replace human thought is going to be some mechanism for live update of weights. "Learning" by injecting into context just isn't good enough. But billions of dollars are going towards research to find these breakthroughs, so we'll get there eventually.

The AI doesn't actually go to sleep at night, it's [mechanistic explanation]

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#140

Post by hypfer »

pjerem wrote: post #136
JohnMakin wrote: post #35

Earlier quoted context omitted.

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…

To me it’s evident that we are a few years away from the Her movie, where everyone on the street is talking to its IA friend. I’m really afraid that it will totally destruct what is remaining of social tissue because why search for friends when you have an always on virtual (and pretty smart) friend h24 in your earbuds ? I’m not blaming anyone for this outcome. I have myself argued with Claude more than once, and rea…

I wouldn't worry about that too much. I would worry about it, but not too much.

People (generally speaking) also eventually stop eating just fastfood. Not all, but many.

So I think we can have some faith in the self-regulation of others.

Post reply on HN