Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

51–60 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#51

Post by danaris »

alexey-salmin wrote: post #45
JohnMakin wrote: post #35

Earlier quoted context omitted.

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…

> It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently. I know people who are like that too. I'm not sure anthropomorphizing is a problem. Seeing analogies everywhere is an innate human trait, sometimes it can be harmful but more often it's useful.

> I know people who are like that too.

This is part of the problem being described. You are part of the problem.

"Some people are bad at X" is not comparable—is not even in the same category—as "LLMs are fundamentally incapable of X".

Every human (at least to a first approximation) is capable of understanding, of learning, of remembering things, of doing math, of counting the number of "r"s in "strawberry".

What you are observing is that some humans are careless, do not take the time and effort to understand, or have internalized the idea that they're "not smart enough" or "not the type of person" who understands things like .

That has nothing remotely to do with the fact that LLMs have no consciousness, no self-awareness, no cognition, no understanding. At a fundamental level.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#52

Post by JohnMakin »

danaris wrote: post #51
alexey-salmin wrote: post #45

Earlier quoted context omitted.

> It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently. I know people who are like that too. I'm not sure anthropomorphizing is a problem. Seeing analogies everywhere is an innate human trait, sometimes it can be harmful but more often it's useful.

> I know people who are like that too. This is part of the problem being described. You are part of the problem. "Some people are bad at X" is not comparable—is not even in the same category —as "LLMs are fundamentally incapable of X". Every human (at least to a first approximation) is capable of understanding, of learning, of remembering things, of doing math, of counting the number of "r"s in "strawberry". What you…

I don't think anyone in this conversation is saying this behavior is anything but the fault of the user not understanding how these tools work? This is a weirdly aggressive post.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#53

Post by nostrademons »

JohnMakin wrote: post #48
traverseda wrote: post #46

Earlier quoted context omitted.

Do you have a working definition of "conscious"?

Do you? I’m not sure what you’re getting at.

Not GP, but one of the challenges with debating whether LLMs are "conscious" is that we don't even really know what it means for a human to be "conscious", or even if consciousness is experienced by other humans the same way it is for ourselves.

What we do know: neurons carry electrical impulses across their synapses to trigger other neurons to fire, and more frequently used synapses are strengthened while infrequently used ones are pruned. This is not all that dissimilar to how a multi-layer perceptron is trained: it's floating point numbers in a big matrix rather than biological structures and electrical impulses, but there is still that element of frequently used connections being strengthened and infrequently used ones being pruned.

What we hypothesize but do not know: there is a thin brain structure of grey matter called the claustrum that has tendrils that reach into nearly every other brain structure. In many ways, this is similar to the attention mechanism of the transformer architecture. It is hypothesized that this may be the seat of consciousness, owing to experiments where electrical stimulation of the claustrum caused patients to immediately lose consciousness. However, there is no way to prove this, owing to the difficulty of otherwise removing or disabling the most connected structure in the brain and observing its effect on consciousness without permanently killing the patient.

Beyond that, we don't know much. I've got a family friend that's been a practicing therapist for 50 years, and I asked him what was the most interesting observation he made in his career. It was that "Everybody experiences the world in a different way, and yet everybody assumes that everyone else experiences the world the same way they do."

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#54

Post by danaris »

JohnMakin wrote: post #52
danaris wrote: post #51

Earlier quoted context omitted.

> I know people who are like that too. This is part of the problem being described. You are part of the problem. "Some people are bad at X" is not comparable—is not even in the same category —as "LLMs are fundamentally incapable of X". Every human (at least to a first approximation) is capable of understanding, of learning, of remembering things, of doing math, of counting the number of "r"s in "strawberry". What you…

I don't think anyone in this conversation is saying this behavior is anything but the fault of the user not understanding how these tools work? This is a weirdly aggressive post.

This is an extremely common fallacy I've seen lots and lots of people fall into with respect to LLMs. In nearly every case, they use the fact that "some humans can't do X" to claim that LLMs are, in fact, basically conscious/human-like/AGI already.

This is deeply untrue, and is highly likely to lead them to bad conclusions about what we can and should do with LLMs.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#55

Post by twothreeone »

While I tend to agree on the overall sentiment, I think this rebuke is inaccurate. Some of these "reasoning" models are trained using "Chain-of-Thought" where the model is presented explicit, intermediate reasoning steps (either by a human or some automation) that supposedly get it closer to the correct answer. These intermediate steps are what was originally called "thinking traces" - not what the model produces to mimic them.

But yes, anthropomorphizing model outputs leads to worse outcomes.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#56

Post by red75prime »

trs83 wrote: post #50
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

Simplifying terminology is not a problem. The providers intentionally choosing terminology to make people think it's something it's not is a problem. I hate the term agent. Calling them companions as some do is just gross.

When I was taking an MIT AI course (in ancient pre-LLM times), an autonomous agent was defined as a system that perceives its environment and acts on it (we were focusing on reward-expectation-maximizing agents, but it's not that important). Peter Norvig has said something like, technically, anything can be described as an agent (a rock maximizes the "follow physical laws" objective), but naturally, it doesn't make much sense to model a rock as an agent. With AI agents, the situation is significantly less controversial: they do perceive, deliberate, and act.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#57

Post by traverseda »

JohnMakin wrote: post #48
traverseda wrote: post #46

Earlier quoted context omitted.

Do you have a working definition of "conscious"?

Do you? I’m not sure what you’re getting at.

You are saying the AI doesn't a some property that you don't have any definition for, not even a working definition. People will disagree on whether a cat or a baby is conscious, they're not debating what a baby or cat is. They're debating this term. You might as well be debating whether an AI is a blorb or not, you have just as good a working definition of blorb as consciousness.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#58

Post by ergl »

kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

> Is anthropomorphizing a real problem?

The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusting wrong answers, if the thinking traces appear convincing enough. Researchers might inspect these traces to try to determine the “intent” of a model, as well.

For an example of the latter, when OpenAI spoke about the hacking of HuggingFace at Black Hat, they repeatedly showed the thinking traces of their model as “proof” of what the model was “thinking” as it performed the attack, calling out “surprise” moments, etc.

Now, it’s possible that the employees presenting didn’t truly believe that the thinking traces would give them useful clues, and presented them only for a “wow” factor, but I wouldn’t discount the possibility that even the people working at frontier companies can fall for this tendency to anthropomorphize LLMs.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#59

Post by red75prime »

JohnMakin wrote: post #35
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…

> if it could, you arguing with it isn't going to make it "learn" or act differently.

Are you talking about a specific harness that doesn't have context retention mechanisms? For example, ChatGPT with disabled memory feature? Or in general where "it" is a fixed-weights network? The latter is trivially true, of course.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#60

Post by adventured »

There's nothing special about 'natural' intelligence as opposed to 'artificial' intelligence, such that we need to concern ourselves with anthropomorphizing mattering any longer.

Those days are over. The age of the classical human has already ended, the species just tends to lag in awareness. The only thing that matters going forward is whether an output makes sense, is it what it should be. Do answers make sense given the context. It doesn't matter if it comes from natural or artificial intelligence.

What I mean is, artificial intelligence is as valid as human intelligence. There's nothing particularly important or special about human feelings or thoughts or memories.

The average human is drastically less important, interesting, intelligent than the latest frontier AI.

Go spend a few years working in retail, you'll quickly understand how absolutely vile humans are on average. Frankly, the reason we should avoid anthropomorphizing AI, is because it's beneath modern AI to mimic something so crude as a human.

Post reply on HN