Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

161–170 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#161

Post by [deleted] »

randomImmigrant wrote: post #98
throw310822 wrote: post #95

Earlier quoted context omitted.

> And it should not go this way, is the OPs point Well, he's wrong. If you argue that LLMs and humans are fundamentally different because all LLMs do X and no human does it, then showing you that it's not true demolishes your argument. Doesn't prove anything positive, but it certainly proves that your argument is invalid.

Well this requires you to buy the very bullshit argument that comparisons of two physically distinct systems just because they share outputs is meaningful. If I call a lightbulb an artificial star, the onus is on me to show the behavior under the hood is star like, not just to point at the light and say “you must see it’s a a star since it’s emitting light!”.

[deleted]

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#162

Post by Dilettante_ »

throw310822 wrote: post #155
randomImmigrant wrote: post #98

Earlier quoted context omitted.

Well this requires you to buy the very bullshit argument that comparisons of two physically distinct systems just because they share outputs is meaningful. If I call a lightbulb an artificial star, the onus is on me to show the behavior under the hood is star like, not just to point at the light and say “you must see it’s a a star since it’s emitting light!”.

> Well this requires you to buy the very bullshit argument Sorry, no. The only thing it requires you to buy is basic logic. If you argue that B is true because of A, the fact that A is false invalidates your argument (I repeat: not B but your argument). There is no question about it.

"One man's modus ponens..."

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#163

Post by ACCount37 »

LLMs are trained on vast bodies of human-curated text that was made by humans or for humans - often both. Then they're tuned further on human feedback. And then they're pointed at tasks humans find to be useful, evaluated by how good they are at those tasks, and trained to get better at them.

You aren't anthropomorphizing LLMs enough.

Are LLM reasoning traces always faithful? Lmao no. Are human inner monologues always faithful? Lmao no. Both of them reflect thoughts somewhat, sometimes. Even in humans, conscious thought is the top of a vast iceberg of subconscious data processing.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#164

Post by throw310822 »

clhodapp wrote: post #87
throw310822 wrote: post #80

Earlier quoted context omitted.

It's also interesting because in humans the existence of "Aha!" moments that are not preceded by or are only loosely related to a chain of thought is taken as the proof of the fundamental mystery and irreproducibility of human intelligence. Now the same argument is made to deny that LLMs actually think. Go figure.

It feels apparent to me that LLMs don't do what is colloquially thought of as thinking. What is less apparent is that humans do .

> What is less apparent is that humans do.

This seems indeed one obvious hole in the argument of the paper. There is no indication whatsoever that human thinking process is more reliable than LLMs intermediate tokens. Which doesn't make our thinking useless, as messy as it might be. We reorder and explain it after the fact.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#165

Post by discreteevent »

yreg wrote: post #86
FuckButtons wrote: post #85

Earlier quoted context omitted.

I would argue, that the null hypothesis is that it is not, and that anyone claiming that there is a mote of consciousness are the ones with the burden of proof.

The null hypothesis is that we don't know jack shit about consciousness. Any claim of certainty seems extraordinary to me and I want to hear the evidence.

Would you want evidence that your your bicycle is not conscious? It can sense your inputs. It responds to the environment. The gears do multiplication (reasoning!).

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#166

Post by yreg »

kelnos wrote: post #150
yreg wrote: post #86

Earlier quoted context omitted.

The null hypothesis is that we don't know jack shit about consciousness. Any claim of certainty seems extraordinary to me and I want to hear the evidence.

> we don't know jack shit about consciousness People keep saying this, but it's not true. We know a lot about consciousness. There's a lot we don't know about it, of course, but "jack shit" is wildly incorrect. > I want to hear the evidence. If you believe LLMs are conscious, then the onus is on you to provide evidence of such.

No, the onus is on either claim.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#167

Post by DarmokTanagra »

kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

[dead]

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#168

Post by elar_verole »

hypfer wrote: post #141
Eisenstein wrote: post #137

Earlier quoted context omitted.

Why can't it be a being? Why is thinking about such a possibilty bad?

Sorry, but if you're genuinely asking this not for trolling reasons then you should _really_ _really_ see a medical professional (not an insult). Online comment sections are not the correct place to unpack any of this. Which is indeed terminating this comment chain, but for good (and benevolent) reason. Doing anything else other than referring to a trained professional in a controlled context would potentially just f…

I'd argue that your colossal overreaction to a small invitation to discuss a somewhat interesting hypothetical is what warrants medical attention

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#169

Post by red75prime »

knollimar wrote: post #109
red75prime wrote: post #100

Earlier quoted context omitted.

I'm not a big fan of arguments like "it's not the real [human quality], it's [mechanistic explanation]." They lack a part: "because the [human quality] allows us to do X, Y, Z, which is impossible with [this mechanism]." I agree that the relevance of retrieved pieces and the management of long-term storage could be improved, though.

It just acts fundamentally different than someone who would remember. If someone only remembered vague scraps of what you'd expect them to remember, you might say the person can't remember. It's much closer to notetaking and reviewing before responding than it is memory. The issue with anthropomorphizing like this is that "memory" comes with baggage of expectations for it to do certain things, and it breaks them. Jus…

> Just like "thinking" implies chain of thought, but you'll frequently get those reasoning traces and then a 180 in the final message.

Be me, think extensively about an exam question, do 180 because the most probable option is just too obvious to be true (no, this part wasn't verbalized, it's how I describe what I felt about making the decision to do 180, or maybe it's a rationalization and it was a natural analog of an unfortunate sample from a probability distribution).

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#170

Post by yreg »

randomImmigrant wrote: post #94
yreg wrote: post #86

Earlier quoted context omitted.

The null hypothesis is that we don't know jack shit about consciousness. Any claim of certainty seems extraordinary to me and I want to hear the evidence.

We know quite a lot about consciousness. You may not, but we know enough to know the informational dynamics in a brain are vastly different from an LLMs. Exactly what kind of certainty are you looking for? Happy to provide it at a molecular, cellular, tissue or whole brain level.

What causes consciousness? Why aren't we all p-zombies?
Post reply on HN