Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

121–130 of 279 posts

Re: Reasoning models don't always say what they think

#121

Post by guerrilla »

kelseyfrog wrote: post #73
kurthr wrote: post #23

Earlier quoted context omitted.

Yes, but to be fair we're much closer to rationalizing creatures than rational ones. We make up good stories to justify our decisions, but it seems unlikely they are at all accurate.

It's even worse - the more we believe ourselves to be rational, the bigger blind spot we have for our own rationalizing behavior. The best way to increase rationality is to believe oneself to be rationalizing! It's one of the reasons I don't trust bayesians who present posteriors and omit priors. The cargo cult rigor blinds them to their own rationalization in the highest degree.

Any links to the research on this?

Re: Reasoning models don't always say what they think

#122

Post by guerrilla »

bluefirebrand wrote: post #69
kurthr wrote: post #23

Earlier quoted context omitted.

Yes, but to be fair we're much closer to rationalizing creatures than rational ones. We make up good stories to justify our decisions, but it seems unlikely they are at all accurate.

I would argue that in order to rationalize, you must first be rational Rationalization is an exercise of (abuse of?) the underlying rational skill

That would be more aesthetically pleasing, but that's unfortunately not what the word rationalizing means.

Re: Reasoning models don't always say what they think

#123

Post by wgd »

nodja wrote: post #35

I highly suspect that CoT tokens are at least partially working as register tokens. Have these big LLM trainers tried replacing CoT with a similar amount of register tokens and see if the improvements are similar?

I remember there was a paper a little while back which demonstrated that merely training a model to output "........" (or maybe it was spaces?) while thinking provided a similar improvement in reasoning capability to actual CoT.

Re: Reasoning models don't always say what they think

#124

Post by pixl97 »

bluefirebrand wrote: post #69
kurthr wrote: post #23

Earlier quoted context omitted.

Yes, but to be fair we're much closer to rationalizing creatures than rational ones. We make up good stories to justify our decisions, but it seems unlikely they are at all accurate.

I would argue that in order to rationalize, you must first be rational Rationalization is an exercise of (abuse of?) the underlying rational skill

Being rational in many philosophical contexts is considered being consistent. Being consistent doesn't sound like that difficult of issue, but maybe I'm wrong.

Re: Reasoning models don't always say what they think

#125

Post by voidspark »

dontlikeyoueith wrote: post #115
voidspark wrote: post #80

Earlier quoted context omitted.

But that's exactly what these deep neural networks have shown, countless times. LLM's generalize on new data outside of its training set. It's called "zero shot learning" where they can solve problems that are not in their training set. AlphaGo Zero is another example. AlphaGo Zero mastered Go from scratch, beating professional players with moves it was never trained on > Another is the fundamental inability to self…

This comment is such a confusion of ideas its comical.

[flagged]

Re: Reasoning models don't always say what they think

#126

Post by TimorousBestie »

justonenote wrote: post #109
no_wizard wrote: post #100

Earlier quoted context omitted.

Because whats inside our minds is more than mathematics, or we would be able to explain human behavior with the purity of mathematics, and so far, we can't. We can prove the behavior of LLMs with mathematics, because its foundations are constructed. That also means it has the same limits of anything else we use applied mathematics for. Is the broad market analysis that HFT firms use software for to make automated tra…

I mean some people have a definition of intelligence that includes a light switch, it has an internal state, it reacts to external stimuli to affect the world around it, so a light switch is more intelligent than a rock. Leaving aside where you draw the line of what classifies as intelligence or not , you seem to be invoking some kind of non-materialist view of the human mind, that there is some other 'essence' that…

No, you don’t need to reach for non-materialistic views in order to conclude that we don’t have a mathematical model (in the sense that we do for an LLM) for how the human brain thinks.

We understand neuron activation, kind of, but there’s so much more going on inside the skull-neurotransmitter concentrations, hormonal signals, bundles with specialized architecture-that doesn’t neatly fit into a similar mathematical framework, but clearly contributes in a significant way to whatever we call human intelligence.

Re: Reasoning models don't always say what they think

#127

Post by miven »

xg15 wrote: post #113
miven wrote: post #107

Earlier quoted context omitted.

I'm not sure I understand what you're trying to say here, information between tokens is propagated through self-attention, and there's an attention block inside each transformer block within the model, that's a whole lot of internal state that's stored in (mostly) inscrutable key and value vectors with hundreds of dimensions per attention head, around a few dozen heads per attention block, and around a few dozen bloc…

Yes, but all that internal state only survives until the end of the computation chain that predicts the next token - it doesn't survive across the entire sequence as it would in a recurrent network. There is literally no difference between a model predicting the tokens " I think the second choice looks best " and a user putting those tokens into the prompt: The input for the next round would be exactly the same. So t…

The key and value vectors are cached, that's kind of the whole point of autoregressive transformer models, the "state" not only survives within the KV cache but, in some sense, grows continuously with each token added, and is reused for each subsequent token.

Re: Reasoning models don't always say what they think

#128

Post by davrosthedalek »

no_wizard wrote: post #100
justonenote wrote: post #78

Earlier quoted context omitted.

> it all rests on (relatively) simple mathematics. We know this is true. We also know that means it has limitations and can't actually reason information. What do you imagine is happening inside biological minds that enables reasoning that is something different to, a lot of, "simple mathematics"? You state that because it is built up of simple mathematics it cannot be reasoning, but this does not follow at all, unle…

Because whats inside our minds is more than mathematics, or we would be able to explain human behavior with the purity of mathematics, and so far, we can't. We can prove the behavior of LLMs with mathematics, because its foundations are constructed. That also means it has the same limits of anything else we use applied mathematics for. Is the broad market analysis that HFT firms use software for to make automated tra…

Your first sentence is a non-sequitur. The fact that we can't explain human behavior does not mean that our minds are more than mathematics.

While absence of proof is not proof of absence, as far as I know, we have not found a physics process in the brain that is not computable in principle.

Re: Reasoning models don't always say what they think

#129

Post by pixl97 »

semiquaver wrote: post #101
hnuser123456 wrote: post #27

Earlier quoted context omitted.

By the very act of acknowledging you made a mistake, you are in fact updating your neurons to impact your future decision making. But that is flat out impossible the way LLMs currently run. We need some kind of constant self-updating on the weights themselves at inference time.

Humans have short term memory. LLMs have context windows. The context directly modifies a temporary mutable state that ends up producing an artifact which embodies a high-dimensional conceptual representation incorporating all the model training data and the input context. Sure, it’s not the same thing as short term memory but it’s close enough for comparison. What if future LLMs were more stateful and had context wi…

Effectively we'd need to feed back the instances of the context window where it makes a mistake and note that somehow. Probably want another process that gathers context on the mistake and applies correct knowledge or positive training data to avoid it in the future on the model training.

Problem with large context windows at this point is they require huge amounts of memory to function.

Re: Reasoning models don't always say what they think

#130

Post by jampekka »

svachalek wrote: post #84
Marazan wrote: post #61

Earlier quoted context omitted.

We had Markov Chains already. Fancy Markov Chains don't seem like a trillion dollar business or actual intelligence.

An LLM is no more a fancy Markov Chain than you are. The math is well documented, go have a read.

About everything can be modelled with large enough Markov Chain, but I'd say stateless autoregressive models like LLMs are a lot easier analyzed as Markov Chains than recurrent systems with very complex internal states like humans.
Post reply on HN