Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

91–100 of 279 posts

Re: Reasoning models don't always say what they think

#91

Post by ianbutler »

lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

https://www.anthropic.com/research/tracing-thoughts-language...

This article counters a significant portion of what you put forward.

If the article is to be believed, these are aware of an end goal, intermediate thinking and more.

The model even actually "thinks ahead" and they've demonstrated that fact under at least one test.

Re: Reasoning models don't always say what they think

#92

Post by tsunamifury »

lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

This type of response is from the typical example of an air chair expert that wildly overestimates their own rationalism and deterministic thinking

Re: Reasoning models don't always say what they think

#94

Post by no_wizard »

highfrequency wrote: post #55
no_wizard wrote: post #48

Earlier quoted context omitted.

That's not at all on par with what I'm saying. There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior. We shouldn't seek to muddy this. EDIT: Generally its accepted that a core trait of intelligence is an agent’s ability to achieve goals in a wide range of environments. This means you must be able to generalize, which in turn allows intelligent beings to react to…

What is that baseline threshold for intelligence? Could you provide concrete and objective results , that if demonstrated by a computer system would satisfy your criteria for intelligence?

see the edit. boils down to the ability to generalize, LLMs can't generalize. I'm not the only one who holds this view either. Francois Chollet, a former intelligence researcher at Google also shares this view.

Re: Reasoning models don't always say what they think

#95

Post by richardw »

One thing I think I’ve found is: reasoning models get more confident and that makes it harder to dislodge a wrong idea.

It feels like I only have 5% of the control, and then it goes into a self-chat where it thinks it’s right and builds on it’s misunderstanding. So 95% of the outcome is driven by rambling, not my input.

Windsurf seems to do a good job of regularly injecting guidance so it sticks to what I’ve said. But I’ve had some extremely annoying interactions with confident-but-wrong “reasoning” models.

Re: Reasoning models don't always say what they think

#96

Post by svachalek »

pton_xd wrote: post #21

I was under the impression that CoT works because spitting out more tokens = more context = more compute used to "think." Using CoT as a way for LLMs "show their working" never seemed logical, to me. It's just extra synthetic context.

This is an interesting paper, it postulates that the ability of an LLM to perform tasks correlates mostly to the number of layers it has, and that reasoning creates virtual layers in the context space. https://arxiv.org/abs/2412.02975

Re: Reasoning models don't always say what they think

#97

Post by justonenote »

hnuser123456 wrote: post #14
lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

When we get to the point where a LLM can say "oh, I made that mistake because I saw this in my training data, which caused these specific weights to be suboptimal, let me update it", that'll be AGI. But as you say, currently, they have zero "self awareness".

We have messed up the terms.

We already have AGI, artificial general intelligence. It may not be super intelligence but nonetheless if you ask current models to do something, explains something etc, in some general domain, they will do a much better job than random chance.

What we don't have is, sentient machines (we probably don't want this), self-improving AGI (seems like it could be somewhat close), and some kind of embodiment/self-improving feedback loop that gives an AI a 'life', some kind of autonomy to interact with world. Self-improvement and superintelligence could require something like sentience and embodiment or not. But these are all separate issues.

Re: Reasoning models don't always say what they think

#98

Post by anon373839 »

dTal wrote: post #52
lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

>The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM Was it ever seriously entertained? I thought the point was not to reveal a chain of thought, but to produce one. A single token's inference must happen in constant time. But an arbitrarily long chain of tokens can encode an arbitrarily complex chain of reasoning. An LLM is e…

> Was it ever seriously entertained?

Yes! By Anthropic! Just a few months ago!

https://www.anthropic.com/research/alignment-faking

Re: Reasoning models don't always say what they think

#99

Post by tibbar »

dingnuts wrote: post #82
byearthithatius wrote: post #57

Earlier quoted context omitted.

"There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior" not really. The whole point they are trying to make is that the capability of these models IS ALREADY muddying the definition of intelligence. We can't really test it because the distribution its learned is so vast. Hence why he have things like ARC now. Even if its just gradient descent based distribution l…

How does an LLM muddy the definition of intelligence any more than a database or search engine does? They are lossy databases with a natural language interface, nothing more.

Ah, but what is in the database? At this point it's clearly not just facts, but problem-solving strategies and an execution engine. A database of problem-solving strategies which you can query with a natural language description of your problem and it returns an answer to your problem... well... sounds like intelligence to me.

Re: Reasoning models don't always say what they think

#100

Post by no_wizard »

justonenote wrote: post #78
no_wizard wrote: post #62

Earlier quoted context omitted.

It’s easy to attribute intelligence these systems. They have a flexibility and unpredictability that hasn't typically been associated with computers, but it all rests on (relatively) simple mathematics. We know this is true. We also know that means it has limitations and can't actually reason information. The corpus of work is huge - and that allows the results to be pretty striking - but once you do hit a corner wit…

> it all rests on (relatively) simple mathematics. We know this is true. We also know that means it has limitations and can't actually reason information. What do you imagine is happening inside biological minds that enables reasoning that is something different to, a lot of, "simple mathematics"? You state that because it is built up of simple mathematics it cannot be reasoning, but this does not follow at all, unle…

Because whats inside our minds is more than mathematics, or we would be able to explain human behavior with the purity of mathematics, and so far, we can't.

We can prove the behavior of LLMs with mathematics, because its foundations are constructed. That also means it has the same limits of anything else we use applied mathematics for. Is the broad market analysis that HFT firms use software for to make automated trades also intelligent?

Post reply on HN