Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

arxiv.org

Post reply on HN
Print view View original

71–80 of 296 posts

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#71

Post by dagss »

I can agree that not calling it "reasoning" may be correct.

But who knows what human "thinking" is really about. If I find a solution to something it is seldom by painstakingly tracing that A and B leads to C (for that I'd need pen and paper). Rather, thoughts just swirl around and then suddenly a solution, or a hunch about a direction to go in, pops into my mind. Who knows what such thoughts "look like" in humans. It is not all of it I can introspect.

Yes I can sort of follow along some kind of train of thought in my head, but there's a lot going on between each thing I'm consciously aware of that I'm not aware of at all, which probably dominates what you are consciously aware of. (Humans are experts at post-rationalization and so on.)

I see this pattern a lot in AI anthro discussions: (1) Assume humans are some kind of perfect idealistic reasonable beings. (2) Hold LLMs up to the standard of an perfect idealistic reasonable being. (3) Conclude that LLMs fails this test, and are therefore not "intelligent", or in this case "thinking", like humans are.

Problem with the argument is comparing humans in anyway to something that is idealistic, reasonable, intelligent in the sense that is implied in these discussions. Human minds are a mess too and fall short of the same standards, just in very different ways from LLMs.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#72

Post by vidarh »

dagss wrote: post #71

I can agree that not calling it "reasoning" may be correct. But who knows what human "thinking" is really about. If I find a solution to something it is seldom by painstakingly tracing that A and B leads to C (for that I'd need pen and paper). Rather, thoughts just swirl around and then suddenly a solution, or a hunch about a direction to go in, pops into my mind. Who knows what such thoughts "look like" in humans. I…

Indeed, except for in the rare cases we painstakingly trace externalised logic we have zero evidence that humans verbalised explanations of our reasoning matches our internal states either, and plenty of evidence via Sperry's split brain experiments that we're prone to outright making up rationalisations for our reasoning.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#73

Post by wahern »

DoctorOetker wrote: post #61

Peculiarly vocal, where were all these people when they started calling the machines computers, anthropomorphizing them akin to the original human (most often female) computers that used to run such calculations? And how dangerous the consequences, we've been dead reckoning for 60-70 years with the wrong terminology without course correction! Where were these vocal people when the "raster-oriented ink deposition mach…

They were there, complaining. You just don't remember them because it's easier for the meaning of a word to shift, or at least take on additional contextual meaning, than it is to get people to use a new word once it's reached critical mass. Those people lost the language fight, but were arguably still vindicated, to the extent they were railing against misguided beliefs that equivocated the capacity of the new machines with their human (or more human-involved) predecessor technologies. Who you also don't remember are the people who made extravagant claims and prognostications based on the equivocation.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#74

Post by brookst »

ergl wrote: post #58
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

> Is anthropomorphizing a real problem? The paper argues that pretending that the so-called thinking traces represent real reasoning can lead users into trusting wrong answers, if the thinking traces appear convincing enough. Researchers might inspect these traces to try to determine the “intent” of a model, as well. For an example of the latter, when OpenAI spoke about the hacking of HuggingFace at Black Hat, they r…

But how is that any different than people being misled by real humans saying words that reflect real thinking, but which are actually dead wrong?

The fallacy here is "thinking == correct", not "tokens == thinking"

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#75

Post by brookst »

trs83 wrote: post #50
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

Simplifying terminology is not a problem. The providers intentionally choosing terminology to make people think it's something it's not is a problem. I hate the term agent. Calling them companions as some do is just gross.

All of these terms were picked by individuals, years ago, while reaching for metaphors that made sense to them personally.

None of these "agent" / "thinking" / "reasoning" terms were dreamed up in boardrooms to intentionally mislead people. They are useful but faulty metaphors; there is no conspiracy.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#76

Post by throw310822 »

danaris wrote: post #54
JohnMakin wrote: post #52

Earlier quoted context omitted.

I don't think anyone in this conversation is saying this behavior is anything but the fault of the user not understanding how these tools work? This is a weirdly aggressive post.

This is an extremely common fallacy I've seen lots and lots of people fall into with respect to LLMs. In nearly every case, they use the fact that "some humans can't do X" to claim that LLMs are, in fact, basically conscious/human-like/AGI already. This is deeply untrue, and is highly likely to lead them to bad conclusions about what we can and should do with LLMs.

> This is an extremely common fallacy ... they use the fact that "some humans can't do X" to claim that LLMs are, in fact, basically conscious/human-like/AGI already.

Claiming that LLMs are conscious or human-like because humans can't do X seems a very strange way to argue for LLM intelligence.

Usually, it goes the other way around: an LLM sceptic says "LLMs are dumb because they can't do X" and soon someone has to remind them that also most of the population can't, in fact, do X.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#77

Post by Jtarii »

JohnMakin wrote: post #35
kgeist wrote: post #28

Is anthropomorphizing a real problem? From what I know, none of the serious LLM researchers believe it has anything to do with human reasoning, apart from Anthropic with their click-baity terminology like "LLM biology". It's just a metaphor. "Reasoning tokens" is simpler to say than "learned prompt augmentation tokens". I used to (and still do) anthropomorphize things long before LLMs, and I've seen my colleagues do…

Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity, not just some turn of phrase. I have a coworker that spends at least 10 hours a week arguing with his like you would with a conscious person. I've gently tried to explain it's like arguing with your compiler for giving you an incoherent error message - it's pointless. It doesn't understand, it can't understa…

>Because plenty of people, even ones that should know better, really believe it's a conscious, thinking entity

You need evidence to make the positive claim that LLMs do not posses any form of consciousness.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#78

Post by DoctorOetker »

wahern wrote: post #73
DoctorOetker wrote: post #61

Peculiarly vocal, where were all these people when they started calling the machines computers, anthropomorphizing them akin to the original human (most often female) computers that used to run such calculations? And how dangerous the consequences, we've been dead reckoning for 60-70 years with the wrong terminology without course correction! Where were these vocal people when the "raster-oriented ink deposition mach…

They were there, complaining. You just don't remember them because it's easier for the meaning of a word to shift, or at least take on additional contextual meaning, than it is to get people to use a new word once it's reached critical mass. Those people lost the language fight, but were arguably still vindicated, to the extent they were railing against misguided beliefs that equivocated the capacity of the new machi…

were they complaining about terminology, or were they complaining about the prospect of losing their jobs?

I'd be happy to revise my opinion if you can demonstrate similar vocal strength on the terminological aspects for those transitions...

You also shifted the goal posts from qualitative to quantitative performance claims. If we ignore that technologies have multiple figures of merit and pretend it's one dimensional, there is a difference between the claim that the machine isn't "printing" vs the machine isn't "printing as well as a human would".

I don't think any of the human printers in the past exceeded the performance levels of current printing technologies, but surely they did exceed the very first machine printers, every technology gets a foot in the door in some niche, and then progressively captures the initially not-yet-automated skills of machine operators.

Would you say an industrial textile weaving machine doesn't weave? At the end of the day its just automation all over again.

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#79

Post by throw310822 »

smugtrain wrote: post #14

Strong dislike for papers that tell me what to do in the title, especially when even the paper admits a loose correlation of the intermediate tokens compared to solution correctness. My solutions work and they speak for themselves.

I admit I didn't read the paper, but if thinking traces are not "thinking", then what are they? If their content is not representing progress towards a solution then they are irrelevant and we should just be able to remove them and save a lot of time and money. There's a lot of money to be made by doing so. So why are they there at all? What do they represent?

Re: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces (2025)

#80

Post by throw310822 »

clhodapp wrote: post #4

Seems like they are closer to scratch than reasoning... Generating some scratch to draw from helps make it easier to compute the real answer.

It's also interesting because in humans the existence of "Aha!" moments that are not preceded by or are only loosely related to a chain of thought is taken as the proof of the fundamental mystery and irreproducibility of human intelligence. Now the same argument is made to deny that LLMs actually think. Go figure.
Post reply on HN