Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

141–150 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#141

Post by hypfer »

Eisenstein wrote: post #137
hypfer wrote: post #127

Earlier quoted context omitted.

Cyberpsychosis. A next token predictor is not a being. Stop posting these things. Stop thinking these things.

Why can't it be a being? Why is thinking about such a possibilty bad?

Sorry, but if you're genuinely asking this not for trolling reasons then you should _really_ _really_ see a medical professional (not an insult).

Online comment sections are not the correct place to unpack any of this.

Which is indeed terminating this comment chain, but for good (and benevolent) reason. Doing anything else other than referring to a trained professional in a controlled context would potentially just feed delusions, which is highly unethical.

Might not even be yours but those of another reader.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#142

Post by internet_points »

Jeff_Brown wrote: post #5
florianherrengt wrote: post #2

> While a human may say “aha” to indicate exactly a sudden internal state change, this interpretation is unwarranted for models which do not have any such internal state, and which on the next forward pass will only differ from the pre-aha pass by the inclusion of that single token in their context. Interpreting the “aha” moment as meaningful exemplifies the long-neglected assumption about long CoT models – the false…

By itself, "aha" carries no insight, but the insight is probably stated immediately after it. In that case the aha is semantically useful, by identifying the insight it is near.

Ooh so lets just change the initial prompt to

    [old prompt asking for some complicated solution requiring insight]
    
    Aha!
and since Aha! is near the good stuff in the network it will just work =P

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#143

Post by pjerem »

hypfer wrote: post #140
pjerem wrote: post #136

Earlier quoted context omitted.

To me it’s evident that we are a few years away from the Her movie, where everyone on the street is talking to its IA friend. I’m really afraid that it will totally destruct what is remaining of social tissue because why search for friends when you have an always on virtual (and pretty smart) friend h24 in your earbuds ? I’m not blaming anyone for this outcome. I have myself argued with Claude more than once, and rea…

I wouldn't worry about that too much. I would worry about it, but not too much. People (generally speaking) also eventually stop eating just fastfood. Not all, but many. So I think we can have some faith in the self-regulation of others.

Well, I have bad news : https://p.kagi.com/proxy/bja35kxoyryc1.png?c=TklOzPjLPioJ5YM...

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#144

Post by zombot »

I think it's essentially too late for exhortations like this. The Believers™ and The Skeptics™ are two thoroughly separated tribes by now that speak two different languages. The chances of one influencing the other in any significant way are minute in my estimation.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#145

Post by zombot »

pjerem wrote: post #143
hypfer wrote: post #140

Earlier quoted context omitted.

I wouldn't worry about that too much. I would worry about it, but not too much. People (generally speaking) also eventually stop eating just fastfood. Not all, but many. So I think we can have some faith in the self-regulation of others.

Well, I have bad news : https://p.kagi.com/proxy/bja35kxoyryc1.png?c=TklOzPjLPioJ5YM...

That link just runs an infinite reloading loop in my browser. Bad news indeed.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#146

Post by zombot »

DoctorOetker wrote: post #78
wahern wrote: post #73

Earlier quoted context omitted.

They were there, complaining. You just don't remember them because it's easier for the meaning of a word to shift, or at least take on additional contextual meaning, than it is to get people to use a new word once it's reached critical mass. Those people lost the language fight, but were arguably still vindicated, to the extent they were railing against misguided beliefs that equivocated the capacity of the new machi…

were they complaining about terminology, or were they complaining about the prospect of losing their jobs? I'd be happy to revise my opinion if you can demonstrate similar vocal strength on the terminological aspects for those transitions... You also shifted the goal posts from qualitative to quantitative performance claims. If we ignore that technologies have multiple figures of merit and pretend it's one dimensiona…

OK, you can split a hair with your bare hand while blindfolded. Congrats, I guess.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#147

Post by kelnos »

kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

> Is anthropomorphizing a real problem?

Yes. Actual real people believe that LLMs are actual, thinking, intelligences, perhaps even with consciousness.

Your bit with MySQL is harmless because it's obvious that a database isn't a sentient lifeform. But LLMs can look like they're the real deal, and people believe it is. Using terminology like "thinking" and "reasoning" to describe what they do only reinforces this.

Having said that, I agree with you on the terminology front: I'm not going to say "learned prompt augmentation tokens" either.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#148

Post by watwut »

randomImmigrant wrote: post #92
dataviz1000 wrote: post #67

Although I 100% agree that the core mechanism of GRPO is purely mechanical token-by-token probability generation, because RL only rewards exact final answers, the training forces the model to develop error-correction habits. This makes the output extremely like human thinking when solving a problem. It's like the order of the thinking tokens is what causes it to get that sweet, delicious reward, and this order seems…

“ This makes the output extremely like human thinking when solving a problem.” This sounds a little like someone saying a lightbulbs output is extremely like the output of stellar fusion. In one sense, yes. Bulbs are in fact designed to take over when our nearest star is beyond the horizon. But that really doesn’t mean you call the bulbs mini stars.

I suspect, it does not matter whether it is thinking or not. The point is to convince most of us that it is thinking.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#149

Post by Otterly99 »

kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

I actually think on the LLM side it might be beneficial to refer to them as because it explicitely guide the token generation towards a "thinking space".

As weird as it is, anthropomorphizing LLMs in prompts has been actually pretty useful (think of the latest big math discoveries which were achieved by having the user giving supporting words). It would be interesting to see if a LLM would perform worse if you used a more neutral term.

The paper's argument is rather than using terms like "thinking trace" can lead people to believe that the model is really thinking, and thus these traces can be used as a sort of interpratbility parameter. This can give a false sense of security when building a LLM-based system which requires guardrails and tracability.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#150

Post by kelnos »

yreg wrote: post #86
FuckButtons wrote: post #85

Earlier quoted context omitted.

I would argue, that the null hypothesis is that it is not, and that anyone claiming that there is a mote of consciousness are the ones with the burden of proof.

The null hypothesis is that we don't know jack shit about consciousness. Any claim of certainty seems extraordinary to me and I want to hear the evidence.

> we don't know jack shit about consciousness

People keep saying this, but it's not true. We know a lot about consciousness. There's a lot we don't know about it, of course, but "jack shit" is wildly incorrect.

> I want to hear the evidence.

If you believe LLMs are conscious, then the onus is on you to provide evidence of such.

Post reply on HN