Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

121–130 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#121

Post by randomImmigrant »

lern_too_spel wrote: post #118
randomImmigrant wrote: post #92

Earlier quoted context omitted.

“ This makes the output extremely like human thinking when solving a problem.” This sounds a little like someone saying a lightbulbs output is extremely like the output of stellar fusion. In one sense, yes. Bulbs are in fact designed to take over when our nearest star is beyond the horizon. But that really doesn’t mean you call the bulbs mini stars.

The visible light produced by both an incandescent bulb and a star is a result of black body radiation, but otherwise, I don't understand your point. A light bulb produces light, something we might have relied on stars to do before. An LLM produces thoughts, something we might have relied on people to do before. Nobody is claiming the process by which the thoughts are produced is the same, only that they both produce…

LLMs produce language. And you're right, if we restricted claims to that, no one would object. It might even be scientifically accurate, shock of shocks.

If you claim LLMs produce thought, it's equivalent to claiming light bulbs undergo fission. Pure wish fulfillment. Language is not the extent of thought, and calling a language producing machine necessarily a thinking machine is an old old mistake.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#122

Post by SpicyLemonZest »

red75prime wrote: post #100
JohnMakin wrote: post #88

Earlier quoted context omitted.

Even claude with “memory” enabled isn’t really “remembering” anything. It just injects it into the context and you hope it happens to find it relevant in its attention mechanisms, and then remembers to actually act on it. Anthropic’s own documentation states claude can and will ignore/truncate these. It’s a context trick, nothing approaching actual “memory,” and in fact, arguing with it will make a bunch of memory fi…

I'm not a big fan of arguments like "it's not the real [human quality], it's [mechanistic explanation]." They lack a part: "because the [human quality] allows us to do X, Y, Z, which is impossible with [this mechanism]." I agree that the relevance of retrieved pieces and the management of long-term storage could be improved, though.

I’m normally not a big fan, but in this particular case it matters a lot. I could come up with some functional argument, but really I care from a model welfare perspective, whether the model understands its reasoning traces to be a part of itself or it’s simply predicting what a character who wrote the current intermediate tokens would output next.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#123

Post by eloisius »

JohnMakin wrote: post #35
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…

Some tools like code rabbit (PR review bot) encourage you to do this. I couldn’t believe I found myself replying to code review comments to explain to an AI why we would rather let an exception crash the app than to catch and hide it several times so that it would stick in its memory. Having to interact with bots as if they are humans, especially when they are gate keeping, is degrading.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#124

Post by taosx »

In the past I managed to get measureable performance optimizing a harness by looking at few traces to see if the traces contained surprised, a lot of text in order to figure out how to use my custom tool, then renamed the tool, changed some parameters and it was already great across around 20 eval tasks in rust/typescript, I repeated the same more recently but I used an llm to look at the traces... didn't achieve the desired result, mostly due to how cost-prohibitive it's for me to run expensive models.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#125

Post by hypfer »

eloisius wrote: post #123
JohnMakin wrote: post #35

Earlier quoted context omitted.

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…

Some tools like code rabbit (PR review bot) encourage you to do this. I couldn’t believe I found myself replying to code review comments to explain to an AI why we would rather let an exception crash the app than to catch and hide it several times so that it would stick in its memory. Having to interact with bots as if they are humans, especially when they are gate keeping, is degrading.

Not just coderabbit (though it is a bad offender). GitHub's copilot review feature is an equally miserable experience. That one loves talking in imperatives, regardless of the fact that it is a clueless machine.

FWIW, this tells you a lot about both the culture behind who built these things but also about the people that enable this stuff and don't immediately nope out. From that perspective, it's a low price to pay to learn whose judgement to never trust again.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#126

Post by hypfer »

alexey-salmin wrote: post #45
JohnMakin wrote: post #35

Earlier quoted context omitted.

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…

> It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently. I know people who are like that too. I'm not sure anthropomorphizing is a problem. Seeing analogies everywhere is an innate human trait, sometimes it can be harmful but more often it's useful.

> I know people who are like that too.

Cool.

Question: Why _on earth_ would we make more of them?

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#127

Post by hypfer »

salawat wrote: post #110
tavavex wrote: post #82

Earlier quoted context omitted.

Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were. Using humanizing language invites discussions of things like the rights and feelings of an algorithm. A judge that is misled by the application of human-centric language to an algorithm can lead to some terrible outcomes. Not everyone is an LLM expert and the language people use leads to them treating LLMs li…

What is terrifying is the propensity of people hoping for a mechanical slave to do everything possible to avoid touching on the possibility for being-ness of the technology they are desperately hoping will work as a basis for that implementation.

Cyberpsychosis. A next token predictor is not a being.

Stop posting these things. Stop thinking these things.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#128

Post by lern_too_spel »

randomImmigrant wrote: post #119
lern_too_spel wrote: post #115

Earlier quoted context omitted.

Why aren't humans simply biological machines? There is no "science" that GP is brushing aside. You need to provide repeatable observations or experiments that GP is ignoring.

Please define "simply biological machines". I'm not sure "biological machine" had a proper definition. What's machine like about biology exactly?

A "machine" is a term that is well defined in science. https://en.wikipedia.org/wiki/Machine

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#130

Post by lern_too_spel »

randomImmigrant wrote: post #120
lern_too_spel wrote: post #114

Earlier quoted context omitted.

Whether human consciousness exists on neural nets or otherwise doesn't disallow an ANN in a particular configuration from being conscious. You might as well argue that human consciousness requires biological neurons, so artificial consciousness can't exist.

Odd. Why don't the details of how consciousness arises in one system inform your judgment of whether it can exist in a different system that only has partial structural overlap? Seems wildly convenient. Where else in science can you show me such a comparable situation in how you define properties?

> Why don't the details of how consciousness arises in one system inform your judgment of whether it can exist in a different system that only has partial structural overlap?

A light bulb doesn't need to do fusion to make light. An airplane doesn't need to flap its wings to fly. An ANN doesn't need to use a brain's structure to think.

Post reply on HN