Reasoning models don't always say what they think • Hacker News Board

Re: Reasoning models don't always say what they think

#111

Post by pixl97 » Thu, Apr 03, 2025, 7:36 PM UTC

no_wizard wrote: post #50

tsimionescu wrote: post #47

Earlier quoted context omitted.

One of the earliest things that defined what AI meant were algorithms like A*, and then rules engines like CLIPS. I would say LLMs are much closer to anything that we'd actually call intelligence, despite their limitations, than some of the things that defined* the term for decades. * fixed a typo, used to be "defend"

>than some of the things that defend the term for decades There have been many attempts to pervert the term AI, which is a disservice to the technologies and the term itself. Its the simple fact that the business people are relying on what AI invokes in the public mindshare to boost their status and visibility. Thats what bothers me about its misuse so much

While it could possibly be to point out so abruptly, you seem to be the walking talking definition of the AI Effect.

>The "AI effect" refers to the phenomenon where achievements in AI, once considered significant, are re-evaluated or redefined as commonplace once they become integrated into everyday technology, no longer seen as "true AI".

Re: Reasoning models don't always say what they think

#112

Post by stevenAthompson » Thu, Apr 03, 2025, 7:37 PM UTC

no_wizard wrote: post #94

highfrequency wrote: post #55

Earlier quoted context omitted.

What is that baseline threshold for intelligence? Could you provide concrete and objective results , that if demonstrated by a computer system would satisfy your criteria for intelligence?

see the edit. boils down to the ability to generalize, LLMs can't generalize. I'm not the only one who holds this view either. Francois Chollet, a former intelligence researcher at Google also shares this view.

> Francois Chollet, a former intelligence researcher at Google also shares this view.

Great, now there are two of you.

Re: Reasoning models don't always say what they think

#113

Post by xg15 » Thu, Apr 03, 2025, 7:39 PM UTC

miven wrote: post #107

xg15 wrote: post #75

> There’s no specific reason why the reported Chain-of-Thought must accurately reflect the true reasoning process; Isn't the whole reason for chain-of-thought that the tokens sort of are the reasoning process? Yes, there is more internal state in the model's hidden layers while it predicts the next token - but that information is gone at the end of that prediction pass. The information that is kept "between one token…

I'm not sure I understand what you're trying to say here, information between tokens is propagated through self-attention, and there's an attention block inside each transformer block within the model, that's a whole lot of internal state that's stored in (mostly) inscrutable key and value vectors with hundreds of dimensions per attention head, around a few dozen heads per attention block, and around a few dozen bloc…

Yes, but all that internal state only survives until the end of the computation chain that predicts the next token - it doesn't survive across the entire sequence as it would in a recurrent network.

There is literally no difference between a model predicting the tokens " I think the second choice looks best " and a user putting those tokens into the prompt: The input for the next round would be exactly the same.

So the tokens kind of act like a bottleneck (or more precisely the sampling of exactly one next token at the end of each prediction round does). During prediction of one token, the model can go crazy with hidden state, but not across several tokens. That forces the model to do "long form" reasoning through the tokens and not through hidden state.

Re: Reasoning models don't always say what they think

#114

Post by freehorse » Thu, Apr 03, 2025, 7:39 PM UTC

It is nonsense to take whatever an LLM writes in its CoT too seriously. I try to classify some messy data, writing "if X edge case appears, then do Y instead of Z". The model in its CoT took notice of X, wrote it should do Y and... it would not do it in the actual output.

The only way to make actual use of LLMs imo is to treat them as what they are, a model that generates text based on some statistical regularities, without any kind of actual understanding or concepts behind that. If that is understood well, one can know how to setup things in order to optimise for desired output (or "alignment"). The way "alignment research" presents models as if they are actually thinking or have intentions of their own (hence the choice of the word "alignment" for this) makes no sense.

Re: Reasoning models don't always say what they think

#115

Post by dontlikeyoueith » Thu, Apr 03, 2025, 7:41 PM UTC

voidspark wrote: post #80

no_wizard wrote: post #70

Earlier quoted context omitted.

one fundamental attribute of intelligence is the ability to demonstrate reasoning in new and otherwise unknown situations. There is no system that I am currently aware of that works on data it is not trained on. Another is the fundamental inability to self update on outdated information. It is incapable of doing that, which means it lacks another marker, which is being able to respond to changes of context effectivel…

But that's exactly what these deep neural networks have shown, countless times. LLM's generalize on new data outside of its training set. It's called "zero shot learning" where they can solve problems that are not in their training set. AlphaGo Zero is another example. AlphaGo Zero mastered Go from scratch, beating professional players with moves it was never trained on > Another is the fundamental inability to self…

This comment is such a confusion of ideas its comical.

Re: Reasoning models don't always say what they think

#116

Post by pixl97 » Thu, Apr 03, 2025, 7:42 PM UTC

no_wizard wrote: post #100

justonenote wrote: post #78

Earlier quoted context omitted.

> it all rests on (relatively) simple mathematics. We know this is true. We also know that means it has limitations and can't actually reason information. What do you imagine is happening inside biological minds that enables reasoning that is something different to, a lot of, "simple mathematics"? You state that because it is built up of simple mathematics it cannot be reasoning, but this does not follow at all, unle…

Because whats inside our minds is more than mathematics, or we would be able to explain human behavior with the purity of mathematics, and so far, we can't. We can prove the behavior of LLMs with mathematics, because its foundations are constructed. That also means it has the same limits of anything else we use applied mathematics for. Is the broad market analysis that HFT firms use software for to make automated tra…

>Because whats inside our minds is more than mathematics,

uh oh, this sounds like magical thinking.

What exactly in our mind is "more" than mathematics exactly.

>or we would be able to explain human behavior with the purity of mathematics

Right, because we understood quantum physics right out of the gate and haven't required a century of desperate study to eek more knowledge from the subject.

Unfortunately it sounds like you are saying "Anything I don't understand is magic", instead of the more rational "I don't understand it, but it seems to be built on repeatable physical systems that are complicated but eventually deciperable"

Re: Reasoning models don't always say what they think

#117

Post by jstummbillig » Thu, Apr 03, 2025, 7:43 PM UTC

lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

Ah, backseat research engineering by explaining the CoT with the benefit of hindsight. Very meta.

Re: Reasoning models don't always say what they think

#118

Post by voidspark » Thu, Apr 03, 2025, 7:43 PM UTC

no_wizard wrote: post #94

highfrequency wrote: post #55

Earlier quoted context omitted.

What is that baseline threshold for intelligence? Could you provide concrete and objective results , that if demonstrated by a computer system would satisfy your criteria for intelligence?

see the edit. boils down to the ability to generalize, LLMs can't generalize. I'm not the only one who holds this view either. Francois Chollet, a former intelligence researcher at Google also shares this view.

Chollet's argument was that it's not "true" generalization, which would be at the level of human cognition. He sets the bar so high that it becomes a No True Scotsman fallacy. The deep neural networks are practically generalizing well enough to solve many tasks better than humans.

Re: Reasoning models don't always say what they think

#119

Post by thoughtlede » Thu, Apr 03, 2025, 7:43 PM UTC

It feels to me that the hypothesis of this research was somewhat "begging the question". Reasoning models are trained to spit some tokens out that increase the chance of the models spitting the right answer at the end. That is, the training process is singularly optimizing for the right answer, not the reasoning tokens.

Why would you then assume the reasoning tokens will include hints supplied in the prompt "faithfully"? The model may or may not include the hints - depending on whether the model activations believe those hints are necessary to arrive at the answer. In their experiments, they found between 20% and 40% of the time, the models included those hints. Naively, that sounds unsurprising to me.

Even in the second experiment when they trained the model to use hints, the optimization was around the answer, not the tokens. I am not surprised the models did not include the hints because they are not trained to include the hints.

That said, and in spite of me potentially coming across as an unsurprised-by-the-result reader, it is a good experiment because "now we have some experimental results" to lean into.

Kudos to Anthropic for continuing to study these models.

Re: Reasoning models don't always say what they think

#120

Post by Robin_Message » Thu, Apr 03, 2025, 7:44 PM UTC

ianbutler wrote: post #91

lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

https://www.anthropic.com/research/tracing-thoughts-language... This article counters a significant portion of what you put forward. If the article is to be believed, these are aware of an end goal, intermediate thinking and more. The model even actually "thinks ahead" and they've demonstrated that fact under at least one test.

The weights are aware of the end goal etc. But the model does not have access to these weights in a meaningful way in the chain of thought model.

So the model thinks ahead but cannot reason about it's own thinking in a real way. It is rationalizing, not rational.