Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

> No, it's a modifier. Just as the word "simple" has a meaning separate from "machine," and "simple machine" has another meaning.

Not comparable. Simple and biological are not remotely similar modifiers.

> Why? LLMs will repurpose code written for other purposes on their own.

They will not, unprompted. Every instance of LLMs doing things requires a prompt, somewhere up the chain, or a harness MD file that automatically gives it a prompt.

No LLM has been created that decides on its own to run random text through it's forward pass. If you say they are doing this, please provide evidence.

> Because nobody has tasked an LLM agent to self replicate. This is an AI safety issue, not a technical issue. The reason that one arose naturally is that it takes a lot more machinery to get to an initial self replicating LLM agent, so it is exceedingly unlikely to arise by chance.

I just asked my agent to self replicate. It couldn't. It said it has no mechanism to read its own weights and copy them.

I'm not sure where you're going with this argument. No one asked a cell to self replicate either. And the chance of a self replicating cell arising is indeed astronomically unlikely. Yet here we are, 4 billion years later.

Are you claiming we're just waiting for an LLM to self replicate? If you prompt a local model, by showing it its own weight file, and ask it to copy that, it will. Digital copies are cheap, remember? Yet no one would call this reproduction, replication or any kind of process where evolution can occur.

> That doesn't matter. The way something is built has no bearing on its function.

This is entirely untrue in biology. Another reason your claim that life is just a “biological machine” makes no sense. At least Google (or whatever AI bot you use) first before making these ludicrous claims.

Here’s just one paper you should read to understand how spectacularly wrong you are: https://academic.oup.com/icb/article/61/6/2038/6327548

First line of the abstract: The relationship between structure and function is a major constituent of the rules of life.

>Why should anything be required to be built biologically or additively or subtractively? That's irrelevant to what the produced structure does.

You can’t have it both ways bub. What is thinking, in a strictly biology free context? Can you define it? No? If you reach for biology to define it, then claim you’ve successfully reproduced it in an artificial system, then it’s fair game to ask you to explain the differences.

By your standard, fools gold is gold, and the alchemists achieve Artifical Gold.

No one is claiming cognition, or consciousness or thought can only be done biologically. I am claiming LLMs specifically don’t do these things, and I’ve pointed out here and elsewhere in this discussion why that is.

In response to my asking where the machine got its inner drive, you said “ Evolution selected for organisms that survive long enough to reproduce. Different biological systems handle this differently.”

Yet plants live very long and reproduce all the time. Do they share the inner drive that humans do? Or a dogs?

The reason for the snark is simple: you are de-dimensionalizing a complex biological trait to make a claim that you think supports your point. And have been doing so now for a few turns. If you can flippantly reduce the field I study to only what you understand, but act as if you know more, I’m going to reply with snark.

> Evolution forces goal directed behavior on systems.

It does not. I could choose to not reply to this. I could reply to this in one line. Or I could reply to this in detail. Or I could write up this reply and decide you’re not worth engaging and not submit. The moment I’m writing these words all these options are open to me. Which of these goals is directing my behavior to write about my behavior?

What biology does is allow for agents to set increasingly complex goals and modify them on a whim, either for internally generated reasons or in response to external signals. But an organism can also be utterly aimless. Roll around in bed and do nothing at all, even when it’s feeling anxious about some goals it has coming up. So where’s the forcing in that?

> Of course I do. Do you not understand that hunger and signals and other basic biological signatures are controlled by deterministic pathways?

They are not.

They are controlled by a mix of deterministic and stochastic processes. There is some hardwiring, certainly, but I’m mystified at your belief it’s all hard wired.

You mentioned hunger and compared it to a robot vaccum. So let me use a lobsters stomatogastric ganglion to show you why you’re wrong:

https://pmc.ncbi.nlm.nih.gov/articles/PMC4375070/

Main take home: Experimental and computational studies of small oscillatory circuits reveal that similar rhythms can arise from disparate mechanisms.

The stomatogastric ganglion (just 30 neurons) in crabs and lobsters is a big part of the hunger satiety loop. The main thing is, all the stuff you’d thing add up to deterministic code, like the number of ion channels in a particular cell, can vary widely between individuals. But the ensemble response is, from the outside, similar. And these structural features change in response to hunger and satiety.

Let me put it this way: A robot vacuum has a strict digital partition: it is either cleaning or charging. But when a living creature transitions from hunger to satiety, circulating hormones completely bathe its nervous system, changing the physical properties of its neurons, altering synaptic strengths, and causing the system to restructure itself. The 'code' isn't reacting to a variable; the variable is fundamentally rewriting the machine.

You’ve shifted your goal posts. You asked me for a problem I can solve with thinking. Not a problem another human can solve for me by their thinking. I have given you exactly what you asked for. An LLM cannot solve it. A human can’t solve it for me but can solve it for themselves (or not be capable of it), it varies.

Whatever you thought you were going to prove with your question, I do hope you’ll dwell on what my example tells you about what thinking is. It isn’t a string of words only. It can be that, but it’s more.

> No reasonable definition of thinking requires that thinking can't be paused or turned off. Please state your point more clearly.

It does. Please define thinking however you want. I’ll show you that it’s a requirement.

> Every problem that previously required human thought can be solved by LLM. QED.

Again, this is the same superficial thinking. Firstly, as my example above showed, no. There are other examples. It requires human thought to cook. No LLM can, nor is smell digitizable in an equivalent way to humans, and yet, cooking is one place where having a human nose is determinative to making appetizing food. Your QED is logically absurd, but is also resting on an assertion that’s flatly untrue.

> No. How many times do I have to explain that the mechanism of thinking is not the same, just as the mechanism of producing light is not the same? You need to think to understand this, which is something that frontier LLMs have no problem with, but which most humans struggle with.

When your “proof” for thinking is the output, I will also compare it to the output, light. My analogy shows the idiocy of your argument, and you don’t want to sit with its implications.

LLMs replicate human language. And many tasks human language can do, LLMs can do or help with. Thinking is not just producing word strings. We think about our family. We think about our bodily aches and pains. We think about how best to do various physical tasks. All this we do sans words, and LLMs don’t do. To call their internal dynamics “thinking” is silly wishful thinking by those who don’t have the ability to go read and understand biology.

And it’s a sad statement of how little you think of your own mind that you won’t explore facts that are outside your expertise that will actually teach you how your mind works, and the wondrous complexity underlying it, and the potential it has to keep doing new things.

I’m going to stop engaging at this point. You seem convinced LLMs think. I wish you the joy of that belief, whatever it is. Religion is always absurd, and this new one is no different.

it is a physical system, yes. So is our brain. I do not understand why this pointless fact is given out as an argument. Either explicitly say you think consciousness is supernatural or say what physical elements are missing in your view that makes you feel AI cannot be conscious.