Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

101–110 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#101

Post by ux266478 »

ergl wrote: post #58
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

> Is anthropomorphizing a real problem? The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusting wrong answers, if the thinking traces appear convincing enough. Researchers might inspect these traces to try to determine the “intent” of a model, as well. For an example of the latter, when OpenAI spoke about the hacking of HuggingFace at Black Hat, they r…

FWIW I think the paper's argumentation is extremely weak to begin with. Like in section 4.1, it opens by expressing a sound position of skepticism:

> there are significant questions on whether these traces have any valid semantic import to the end user.

Which it contradicts in the very next paragraph, taking a stance that there are no valid semantics present in the trace:

> the false idea that derivational traces are semantically meaningful

It's really not a high quality paper worth taking seriously.

And that's before we get into the complete and total breakdown of objective analysis. It rejects distributional semantics as a theory, while also explicitly stating the results that have been produced under its auspices are "undeniable". Never elaborated on, and at no point in the paper am I given the impression the authors are even aware of the problem with this. It's just more unempirical slop that wants its pound of flesh without putting the work in. Frankly, whoever let this through peer review should be ashamed of themselves.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#102

Post by Jtarii »

roughly wrote: post #81
Jtarii wrote: post #77

Earlier quoted context omitted.

>Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity You need evidence to make the positive claim that LLMs do not posses any form of consciousness.

Well, our current set of evidence is that it’s a mechanistic mechanical algorithm with an RNG embedded in it and we can both get it to repeatedly produce the same output for the same input and also get it to repeatedly do absolutely nothing at all, which are not characteristics we usually find in objects evincing consciousness. LLMs bear absolutely none of the traits we’ve come to recognize as the external hallmarks…

I would agree that LLMs aren't much like the human brain, that doesn't prove that consciousness is not occurring. Does a fruit fly experience anything? If a microscopic insect can experience something, why can't a CPU?

>if you’re going to go around asserting the LLM is conscious despite all existing evidence to the contrary

Well there is neither any evidence that suggests LLMs are not conscious, and I also never asserted that they are. If I had to guess I would say that any information processing system will produce some kind of conscious experience, but I ultimately have literally no idea.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#103

Post by Jtarii »

randomImmigrant wrote: post #94
yreg wrote: post #86

Earlier quoted context omitted.

The null hypothesis is that we don't know jack shit about consciousness. Any claim of certainty seems extraordinary to me and I want to hear the evidence.

We know quite a lot about consciousness. You may not, but we know enough to know the informational dynamics in a brain are vastly different from an LLMs. Exactly what kind of certainty are you looking for? Happy to provide it at a molecular, cellular, tissue or whole brain level.

Can we conclusively say one way or another if an ant has a conscious experience?

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#104

Post by randomImmigrant »

Jtarii wrote: post #103
randomImmigrant wrote: post #94

Earlier quoted context omitted.

We know quite a lot about consciousness. You may not, but we know enough to know the informational dynamics in a brain are vastly different from an LLMs. Exactly what kind of certainty are you looking for? Happy to provide it at a molecular, cellular, tissue or whole brain level.

Can we conclusively say one way or another if an ant has a conscious experience?

Yes we can say it has conscious experience. The content and depth of it, we cannot yet fully grok, and of course, what it feels like from inside the ant is something we never will know.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#105

Post by JohnMakin »

red75prime wrote: post #100
JohnMakin wrote: post #88

Earlier quoted context omitted.

Even claude with “memory” enabled isn’t really “remembering” anything. It just injects it into the context and you hope it happens to find it relevant in its attention mechanisms, and then remembers to actually act on it. Anthropic’s own documentation states claude can and will ignore/truncate these. It’s a context trick, nothing approaching actual “memory,” and in fact, arguing with it will make a bunch of memory fi…

I'm not a big fan of arguments like "it's not the real [human quality], it's [mechanistic explanation]." They lack a part: "because the [human quality] allows us to do X, Y, Z, which is impossible with [this mechanism]." I agree that the relevance of retrieved pieces and the management of long-term storage could be improved, though.

I just don’t find it really relevant to the argument presented I guess. I disable auto memory and have my own mechanisms and infrastructure with how my agentic system “knows” and “remembers” things which is roughly an automated, sometimes self-correcting working index on the file system. It behaves much better than claude’s automated “memory” system, so I use that, but digging into how that worked and making something of my own just makes me really dismissive of comparing it to something like actual memory, so I apologize if it came off dismissive.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#106

Post by roughly »

Jtarii wrote: post #102
roughly wrote: post #81

Earlier quoted context omitted.

Well, our current set of evidence is that it’s a mechanistic mechanical algorithm with an RNG embedded in it and we can both get it to repeatedly produce the same output for the same input and also get it to repeatedly do absolutely nothing at all, which are not characteristics we usually find in objects evincing consciousness. LLMs bear absolutely none of the traits we’ve come to recognize as the external hallmarks…

I would agree that LLMs aren't much like the human brain, that doesn't prove that consciousness is not occurring. Does a fruit fly experience anything? If a microscopic insect can experience something, why can't a CPU? >if you’re going to go around asserting the LLM is conscious despite all existing evidence to the contrary Well there is neither any evidence that suggests LLMs are not conscious, and I also never asse…

There’s plenty. First, when we say “LLM”, what are we referring to? What is the entity that would be conscious in this case?

The reason this is important is because powerful people are currently trying to use the dodge that LLMs are conscious to launder liability for their own policy choices, so the sloppy thinking and half-assed conjecture about LLM consciousness has real-world consequences, and every time you assert the question is unknowable you allow that kind of loophole, so it’d behoove all of us for you to spend some time actually digging in on this instead of just idly making or rebutting assertions.

There’s a richer literature here than what you’ve seemed to have engaged with, and I’d encourage you to spend some time with it before handing more money to the magic AI people.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#107

Post by JohnMakin »

JohnMakin wrote: post #52
danaris wrote: post #51

Earlier quoted context omitted.

> I know people who are like that too. This is part of the problem being described. You are part of the problem. "Some people are bad at X" is not comparable—is not even in the same category —as "LLMs are fundamentally incapable of X". Every human (at least to a first approximation) is capable of understanding, of learning, of remembering things, of doing math, of counting the number of "r"s in "strawberry". What you…

I don't think anyone in this conversation is saying this behavior is anything but the fault of the user not understanding how these tools work? This is a weirdly aggressive post.

Just a heads up - this person in gp comment, their original post in a way that is much much different than the one I was replying to originally both in tone and content.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#108

Post by JohnMakin »

JohnMakin wrote: post #105
red75prime wrote: post #100

Earlier quoted context omitted.

I'm not a big fan of arguments like "it's not the real [human quality], it's [mechanistic explanation]." They lack a part: "because the [human quality] allows us to do X, Y, Z, which is impossible with [this mechanism]." I agree that the relevance of retrieved pieces and the management of long-term storage could be improved, though.

I just don’t find it really relevant to the argument presented I guess. I disable auto memory and have my own mechanisms and infrastructure with how my agentic system “knows” and “remembers” things which is roughly an automated, sometimes self-correcting working index on the file system. It behaves much better than claude’s automated “memory” system, so I use that, but digging into how that worked and making somethin…

I guess I will expand on what I meant why I react to claude memory acting mechanically or logically anything like human memory, is because it isn’t how memory in the brain works, they’re not comparable.

The layman’s understanding I have of memory, as someone that has dealt with memory issues much of my life, is that memory formation is heavily tied to emotions. emotions are triggered by input which sends a complex set of signals throughout the brain - you’re not just finding where in your head to store this, your brain is deciding how important it is, and what else to correlate it with - so it can tie them to other related memories. then on top of all this, much of the sensory experience you intake is subconsciously compared against high priority memory impressions and deciding what to pay attention to.

you could, argue that the sensory input is the simple md files and the emotional mechanism is the same effect as to how attention mechanisms work in llm’s. Ok, I can almost buy that, but these tools lack a fundamental ability to decide how important things are.

an analogy. you tell a person “if you pick a daisy in the next five years, an assassin will come to kill you” and they hold a knife to your throat while they say it, your brain whether you like it or not is going to say “THIS IS AN IMPORTANT MEMORY I NEVER MUST FORGET” and you’ll see something that looks like a daisy and have a panic attack 3 years later. that memory is never fo tell me claude or other tool harnesses using memory harnesses can prioritize memories the way that human would, instead they forget even when reminded, because the human brain is just so much better at it

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#109

Post by knollimar »

red75prime wrote: post #100
JohnMakin wrote: post #88

Earlier quoted context omitted.

Even claude with “memory” enabled isn’t really “remembering” anything. It just injects it into the context and you hope it happens to find it relevant in its attention mechanisms, and then remembers to actually act on it. Anthropic’s own documentation states claude can and will ignore/truncate these. It’s a context trick, nothing approaching actual “memory,” and in fact, arguing with it will make a bunch of memory fi…

I'm not a big fan of arguments like "it's not the real [human quality], it's [mechanistic explanation]." They lack a part: "because the [human quality] allows us to do X, Y, Z, which is impossible with [this mechanism]." I agree that the relevance of retrieved pieces and the management of long-term storage could be improved, though.

It just acts fundamentally different than someone who would remember.

If someone only remembered vague scraps of what you'd expect them to remember, you might say the person can't remember.

It's much closer to notetaking and reviewing before responding than it is memory.

The issue with anthropomorphizing like this is that "memory" comes with baggage of expectations for it to do certain things, and it breaks them.

Just like "thinking" implies chain of thought, but you'll frequently get those reasoning traces and then a 180 in the final message.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#110

Post by salawat »

tavavex wrote: post #82
alexey-salmin wrote: post #45

Earlier quoted context omitted.

> It doesn't understand, it can't understand, and even if it could, you arguing with it isn't going to make it "learn" or act differently. I know people who are like that too. I'm not sure anthropomorphizing is a problem. Seeing analogies everywhere is an innate human trait, sometimes it can be harmful but more often it's useful.

Anthropomorphizing is a problem when you're talking about treating something that's not living as if it were. Using humanizing language invites discussions of things like the rights and feelings of an algorithm. A judge that is misled by the application of human-centric language to an algorithm can lead to some terrible outcomes. Not everyone is an LLM expert and the language people use leads to them treating LLMs li…

What is terrifying is the propensity of people hoping for a mechanical slave to do everything possible to avoid touching on the possibility for being-ness of the technology they are desperately hoping will work as a basis for that implementation.
Post reply on HN