The mistake is not in saying that the machine thinks. It is in supposing that, because we have used the word "thinks", there must be something in the machine which is the thinking.
Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
261–270 of 296 posts
Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
#262Coming from a more traditional stats/ML background, I try to view "thinking" traces as a way to explore the search space without getting caught in a local maximum. A better analogy for me is annealing; you can't cool metal down instantly or the result is brittle. You must cool down gradually, which allows the molecules to arrange into more durable structures. Random but controlled. In the same way, thinking traces ar…
this seems like more anthropomorphizing. even if it has similar results often enough to be useful, next token prediction is not searching a solution space.
you even end with a paragraph saying “it’s not too dissimilar from what we do”. how is that not anthropomorphizing?
and if what we do isn’t “thinking” then what is or ever has been?
Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
#263Earlier quoted context omitted.
My post «When I am told "beware of the xyphucymmon", I do beware, though not of the xyphucymmon» had meaning, which should be pretty clear. The reply remains: debate is not performance art in which you convey "through the medium of dance and howls". So, yes, communication has obligations for the locutor. So, yes, imperfect speech is a problem - because societies are not performance art arenas. -- @Razengan: speaking…
I mostly agree with your original point. Is it thinking? Is it intelligent? Is it conscious? From my perspective, overloaded words that we’ve reserved to make ourselves feel more special and above other members of the animal kingdom. But you’ve not adequately communicated why you care so much whether anyone labels an LLM as such.
> Is it thinking? Is it intelligent? Is it conscious?
Very different things. "Thinking": "Dijkstra". We try to hire more intelligent people, while we do not have a clear idea on a property of "conscious" which would make hiring preferable.
> to make ourselves feel
Irrational.
> above other members of the animal kingdom
Irrational.
> why you care so much
There should be no hint that I would do specifically.
> whether anyone labels an LLM as such
If anyone labels an LLM , the point of why would that be important should be clear.
And,
> if it hardly means anything why would you be so concerned when it’s uttered? Just ignore it
Because the behaviour you suggest, consistent with the exchanges in an opium parlor, is not behaviour consistent in normal contexts.
Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
#264Coming from a more traditional stats/ML background, I try to view "thinking" traces as a way to explore the search space without getting caught in a local maximum. A better analogy for me is annealing; you can't cool metal down instantly or the result is brittle. You must cool down gradually, which allows the molecules to arrange into more durable structures. Random but controlled. In the same way, thinking traces ar…
this seems like more anthropomorphizing. even if it has similar results often enough to be useful, next token prediction is not searching a solution space. you even end with a paragraph saying “it’s not too dissimilar from what we do”. how is that not anthropomorphizing? and if what we do isn’t “thinking” then what is or ever has been?
> next token prediction is not searching a solution space.
Interesting take. Next token prediction (via the attention mechanism) is a "walk" through the token embedding space. Searching the solution space is what it does, mathematically. It's how we take tokens x context length possible combinations and prune them to converge on viable answers so quickly.
Does it look like search at inference time? No. With given weights, a given prompt, and a given random seed, you get the exact same answer. There's not much searching happening at inference...
The key is that most of that space is searched at training time. The weights implicitly prune the search space, blocking off or make certain token combinations effectively impossible. It's easy to think "we're just applying weights at inference time" without considering all the pre-work that's done to prune that search space.
Which is exactly why "thinking" traces (and randomization) are useful! They bust out of any local optima created by too-tightly-constrained models or system prompts. It's both useful and technically correct to speak of the process as a high-dimension combinatorial search.
Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
#265Earlier quoted context omitted.
>The reason this is important is because powerful people are currently trying to use the dodge that LLMs are conscious to launder liability for their own policy choices, so the sloppy thinking and half-assed conjecture about LLM consciousness has real-world consequences I'm not going to change my beliefs or how I think about interesting questions just because its the "socially conscious" thing to do. >There’s a riche…
You're not engaging with the topic. Engaging with the topic is where one either asks questions and listens to the answer or actually seeks to increase one's knowledge on the topic. You're just saying things. That's lazy. And, you're welcome to do what you want to do, it's your god given right to stay as ignorant as you want about any particular topic, but that comes with consequences. If you want to call that being s…
It's just as intellectually lazy to leave "consciousness" undefined and proceed to claim that "X cannot be consciousness" because of social reasons. That's the definition of a circular argument.
In the same vein, airplanes can't fly because they don't flap their wings.
As for ignorance: have you tried to make an LLM produce the same output for the same input (as you state above)? Give it a shot, you'll be surprised.
Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
#266Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…
> Is anthropomorphizing a real problem? Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness. Your bit with MySQL is harmless because it's obvious that a database isn't a sentient lifeform. But LLMs can look like they're the real deal, and people believe it is. Using terminology like "thinking" and "reasoning" to describe what they do only reinforces this. Hav…
Actual real people think the world is flat, Elvis is alive, and aliens are regularly flying around the planet. Is preventing peoples unimportant personal beliefs really an important goal or is this just more warning label culture?
Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
#267Earlier quoted context omitted.
You're not engaging with the topic. Engaging with the topic is where one either asks questions and listens to the answer or actually seeks to increase one's knowledge on the topic. You're just saying things. That's lazy. And, you're welcome to do what you want to do, it's your god given right to stay as ignorant as you want about any particular topic, but that comes with consequences. If you want to call that being s…
It's just as intellectually lazy to leave "consciousness" undefined and proceed to claim that "X cannot be consciousness" because of social reasons. That's the definition of a circular argument. In the same vein, airplanes can't fly because they don't flap their wings. As for ignorance: have you tried to make an LLM produce the same output for the same input (as you state above)? Give it a shot, you'll be surprised.
I’m not leaving consciousness undefined. I provided a definition and asserted LLMs were not conscious by that definition. I solicited an alternate definition and was given none. If you’d like to claim the LLM is conscious, you need to define what you mean, for both “the LLM” and “is conscious,” because it’s not passing any of our existing bars for consciousness, and the only thing you can point to are characteristics also present in other objects we don’t consider conscious, so again, the burden of proof is in fact on you and that other fellow to provide some definitions here, because as sits you and the other commenter are just saying shit and refusing to engage with any kind of rigor.
And, to your point: I have, and I did, and I was not. If you’d like to refine your argument further - What LLM? Provide input how, and in what fashion? Under what conditions? - we can have that conversation, but if your assertion is “I can’t get ChatGPT to consistently produce the same output twice and therefore it is conscious,” that’s an incredibly facile argument.
You two seem to be laboring under the impression that this is terra ignota philosophically; it’s not. There’s an enormous amount of literature, thinking, ideas, concepts, frameworks, and approaches that already exist here that you’re welcome to engage with, but you’re not doing that, nor are you actually engaging with any of my arguments except to deny their existence.
Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
#268Earlier quoted context omitted.
It's just as intellectually lazy to leave "consciousness" undefined and proceed to claim that "X cannot be consciousness" because of social reasons. That's the definition of a circular argument. In the same vein, airplanes can't fly because they don't flap their wings. As for ignorance: have you tried to make an LLM produce the same output for the same input (as you state above)? Give it a shot, you'll be surprised.
I’m not leaving consciousness undefined. I provided a definition and asserted LLMs were not conscious by that definition. I solicited an alternate definition and was given none. If you’d like to claim the LLM is conscious, you need to define what you mean, for both “the LLM” and “is conscious,” because it’s not passing any of our existing bars for consciousness, and the only thing you can point to are characteristics…
Philosophically - sure. But philosophy is a feedback loop. Humans trying to build a model of their brain within their own brain. There is by definition not enough oomph there, and that model will necessarily be approximate at best.
Think that's what an out of touch techie would say? May I remind you that just a few hundred years ago the best philosophers were debating whether the world would descend into anarchy if more people realized that the big monkey in the sky doesn't exist. Just because philosophers talk about something doesn't mean it exists in reality.
As for same output - there's a parameter called "temperature". It governs the random wandering in the output of an LLM. Reducing temp gives more deterministic output, but also reduces the capabilities. Is it possible that randomness is also precisely the mechanism behind human creativity?
Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
#269Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…
Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…
[deleted]
Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)
#270Earlier quoted context omitted.
> I mean, that is the entire definition of the word. Not quite - my wording there was very deliberate. By saying that it's a problem when you're treating something that's not living (not non-human!) as if it were, that excludes pets and all animals from the equation. I understand how common it is for humans to assign human qualities to other things and beings, but there is also an unspoken variable of intensity. Repr…
No its not unspoken. Read the paper linked. It explains what you think you are explaining but with scientific rigor. And you are wrong in your defintion of the world. Attaching human qualities to any non human entity (living or otherwise) is the accepted defintion of what anthropomorphizing is. It does not only apply to non-living entities.
Now I'm the one who's starting to doubt you read my comment with a fraction of the rigor you demand from me reading that incredibly dense paper.
Read it again. I didn't say the definition of the word didn't include living things. The definition of anything is not discussed there at all. The 'unspoken' part applies to what I left implied in my argument, you don't get to rewrite my argument. What I said is that while anthropomorphization is common and can apply to everything, to me it can be a problem when it:
1. Is applied to non-living things with a genuine conviction they are truly living, and not in a ceremonial or casual way
2. Is particularly intense compared to other cases of anthropomorphization and leads to a desire to assign actual personhood to the object