Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

Post by phoghed » Fri, Aug 21, 2026, 10:50 AM UTC

randomImmigrant wrote: post #92

dataviz1000 wrote: post #67

Although I 100% agree that the core mechanism of GRPO is purely mechanical token-by-token probability generation, because RL only rewards exact final answers, the training forces the model to develop error-correction habits. This makes the output extremely like human thinking when solving a problem. It's like the order of the thinking tokens is what causes it to get that sweet, delicious reward, and this order seems…

“ This makes the output extremely like human thinking when solving a problem.” This sounds a little like someone saying a lightbulbs output is extremely like the output of stellar fusion. In one sense, yes. Bulbs are in fact designed to take over when our nearest star is beyond the horizon. But that really doesn’t mean you call the bulbs mini stars.

Post by discreteevent » Fri, Aug 21, 2026, 10:57 AM UTC

yreg wrote: post #171

discreteevent wrote: post #165

Earlier quoted context omitted.

Would you want evidence that your your bicycle is not conscious? It can sense your inputs. It responds to the environment. The gears do multiplication (reasoning!).

I think that the difference is that there is a vast gap in complexity and abilities, no?

Post by Jtarii » Fri, Aug 21, 2026, 11:02 AM UTC

randomImmigrant wrote: post #104

Jtarii wrote: post #103

Earlier quoted context omitted.

Can we conclusively say one way or another if an ant has a conscious experience?

Yes we can say it has conscious experience. The content and depth of it, we cannot yet fully grok, and of course, what it feels like from inside the ant is something we never will know.

Post by short_sells_poo » Fri, Aug 21, 2026, 11:06 AM UTC

YawningAngel wrote: post #152

kelnos wrote: post #147

Earlier quoted context omitted.

> Is anthropomorphizing a real problem? Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness. Your bit with MySQL is harmless because it's obvious that a database isn't a sentient lifeform. But LLMs can look like they're the real deal, and people believe it is. Using terminology like "thinking" and "reasoning" to describe what they do only reinforces this. Hav…

I think LLMs are pretty clearly intelligent in some sense of the word, and I don't know how one could ever confidently know they aren't conscious in some sense. That's not to say they're humanlike, just that people who think they know these ideas are ridiculous seem to be overreaching in the same way Steve Yegge seems to be overreaching

Post by Jtarii » Fri, Aug 21, 2026, 11:07 AM UTC

roughly wrote: post #106

Jtarii wrote: post #102

Earlier quoted context omitted.

I would agree that LLMs aren't much like the human brain, that doesn't prove that consciousness is not occurring. Does a fruit fly experience anything? If a microscopic insect can experience something, why can't a CPU? >if you’re going to go around asserting the LLM is conscious despite all existing evidence to the contrary Well there is neither any evidence that suggests LLMs are not conscious, and I also never asse…

There’s plenty. First, when we say “LLM”, what are we referring to? What is the entity that would be conscious in this case? The reason this is important is because powerful people are currently trying to use the dodge that LLMs are conscious to launder liability for their own policy choices, so the sloppy thinking and half-assed conjecture about LLM consciousness has real-world consequences, and every time you asser…