Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

131–140 of 279 posts

Re: Reasoning models don't always say what they think

#131

Post by no_wizard »

voidspark wrote: post #80
no_wizard wrote: post #70

Earlier quoted context omitted.

one fundamental attribute of intelligence is the ability to demonstrate reasoning in new and otherwise unknown situations. There is no system that I am currently aware of that works on data it is not trained on. Another is the fundamental inability to self update on outdated information. It is incapable of doing that, which means it lacks another marker, which is being able to respond to changes of context effectivel…

But that's exactly what these deep neural networks have shown, countless times. LLM's generalize on new data outside of its training set. It's called "zero shot learning" where they can solve problems that are not in their training set. AlphaGo Zero is another example. AlphaGo Zero mastered Go from scratch, beating professional players with moves it was never trained on > Another is the fundamental inability to self…

>AlphaGo Zero mastered Go from scratch, beating professional players with moves it was never trained on

Thats all well and good, but it was tuned with enough parameters to learn via reinforcement learning[0]. I think The Register went further and got better clarification about how it worked[1]

>During training, it sits on each side of the table: two instances of the same software face off against each other. A match starts with the game's black and white stones scattered on the board, placed following a random set of moves from their starting positions. The two computer players are given the list of moves that led to the positions of the stones on the grid, and then are each told to come up with multiple chains of next moves along with estimates of the probability they will win by following through each chain.

While I also find it interesting that in both of these instances, its all referenced to as machine learning, not AI, its also important to see that even though what AlphaGo Zero did was quite awesome and a step forward in using compute for more complex tasks, it was still seeded the basics of information - the rules of Go - and simply patterned matched against itself until built up enough of a statistical model to determine the best moves to make in any given situation during a game.

Which isn't the same thing as showing generalized reasoning. It could not, then, take this information and apply it to another situation.

They did show the self reinforcement techniques worked well though, and used them for Chess and Shogi to great success as I recall, but thats a validation of the technique, not that it could generalize knowledge.

>That's an engineering decision, not a fundamental limitation

So you're saying that they can't reason about independently?

[0]: https://deepmind.google/discover/blog/alphago-zero-starting-...

[1]: https://www.theregister.com/2017/10/18/deepminds_latest_alph...

Re: Reasoning models don't always say what they think

#133

Post by [deleted] »

no_wizard wrote: post #70
voidspark wrote: post #40

Earlier quoted context omitted.

You are confusing sentience or consciousness with intelligence.

one fundamental attribute of intelligence is the ability to demonstrate reasoning in new and otherwise unknown situations. There is no system that I am currently aware of that works on data it is not trained on. Another is the fundamental inability to self update on outdated information. It is incapable of doing that, which means it lacks another marker, which is being able to respond to changes of context effectivel…

[deleted]

Re: Reasoning models don't always say what they think

#134

Post by devmor »

johnecheck wrote: post #37
no_wizard wrote: post #17

Earlier quoted context omitted.

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

I don't think your detraction has much merit.

If I don't understand how a combustion engine works, I don't need that engineering knowledge to tell you that a bicycle [an LLM] isn't a car [a human brain] just because it fits the classification of a transportation vehicle [conversational interface].

This topic is incredibly fractured because there is too much monetary interest in redefining what "intelligence" means, so I don't think a technical comparison is even useful unless the conversation begins with an explicit definition of intelligence in relation to the claims.

Re: Reasoning models don't always say what they think

#135

Post by justonenote »

TimorousBestie wrote: post #126
justonenote wrote: post #109

Earlier quoted context omitted.

I mean some people have a definition of intelligence that includes a light switch, it has an internal state, it reacts to external stimuli to affect the world around it, so a light switch is more intelligent than a rock. Leaving aside where you draw the line of what classifies as intelligence or not , you seem to be invoking some kind of non-materialist view of the human mind, that there is some other 'essence' that…

No, you don’t need to reach for non-materialistic views in order to conclude that we don’t have a mathematical model (in the sense that we do for an LLM) for how the human brain thinks. We understand neuron activation, kind of, but there’s so much more going on inside the skull-neurotransmitter concentrations, hormonal signals, bundles with specialized architecture-that doesn’t neatly fit into a similar mathematical…

> it all rests on (relatively) simple mathematics. We know this is true. We also know that means it has limitations and can't actually reason information.

This was the statement I was responding to, it is stating that because it's built on simple mathematics it _cannot_ reason.

Yes we don't have a complete mathematical model of human intelligence, but the idea that because it's built on mathematics that we have modelled, that it cannot reason is nonsensical, unless you subscribe to a non-materialist view.

In a way, he is saying (not really but close) that if we did model human intelligence with complete fidelity, it would no longer be intelligence.

Re: Reasoning models don't always say what they think

#136

Post by bluefirebrand »

guerrilla wrote: post #122
bluefirebrand wrote: post #69

Earlier quoted context omitted.

I would argue that in order to rationalize, you must first be rational Rationalization is an exercise of (abuse of?) the underlying rational skill

That would be more aesthetically pleasing, but that's unfortunately not what the word rationalizing means.

Just grabbing definitions from Google:

Rationalize: "An attempt to explain or justify (one's own or another's behavior or attitude) with logical, plausible reasons, even if these are not true or appropriate"

Rational: "based on or in accordance with reason or logic"

They sure seem like related concepts to me. Maybe you have a different understanding of what "rationalizing" is, and I'd be interested in hearing it

But if all you're going to do is drive by comment saying "You're wrong" without elaborating at all, maybe just keep it to yourself next time

Re: Reasoning models don't always say what they think

#137

Post by xg15 »

miven wrote: post #127
xg15 wrote: post #113

Earlier quoted context omitted.

Yes, but all that internal state only survives until the end of the computation chain that predicts the next token - it doesn't survive across the entire sequence as it would in a recurrent network. There is literally no difference between a model predicting the tokens " I think the second choice looks best " and a user putting those tokens into the prompt: The input for the next round would be exactly the same. So t…

The key and value vectors are cached, that's kind of the whole point of autoregressive transformer models, the "state" not only survives within the KV cache but, in some sense, grows continuously with each token added, and is reused for each subsequent token.

Hmm, maybe I misunderstood that part, but so far I thought the KV cache was really just that - a cache. Because all the previous tokens of the sequence stay the same, it makes no sense to compute the same K and V vectors again in each round.

But that doesn't change that the only input to the Q, K and V calculations are the tokens (or in later layers information that was derived from the tokens) and each vector in the cache maps directly to an input token.

So I think you could disable the cache and recompute everything in each round and you'd still get the same result, just a lot slower.

Re: Reasoning models don't always say what they think

#138

Post by pixl97 »

PeterStuer wrote: post #39

Humans also post-rationalize the things their subconscious "gut feeling" came up with. I have no problem for a system to present a reasonable argument leading to a production/solution, even if that materially was not what happened in the generation process. I'd go even further and pose that probably requiring the "explanation" to be not just congruent but identical with the production would either lead to incomprehen…

Now, at least in a well disciplined human, we can catch when our gut feeling was wrong when the 'create a reasonable argument' process fails. I guess I wonder how well a LLM can catch that and correct it's thinking.

Now I've seen in some models where it figures out it's wrong, but then gets stuck in a loop. I've not really used the larger reasoning models much to see their behaviors.

Re: Reasoning models don't always say what they think

#139

Post by travisjungroth »

no_wizard wrote: post #70
voidspark wrote: post #40

Earlier quoted context omitted.

You are confusing sentience or consciousness with intelligence.

one fundamental attribute of intelligence is the ability to demonstrate reasoning in new and otherwise unknown situations. There is no system that I am currently aware of that works on data it is not trained on. Another is the fundamental inability to self update on outdated information. It is incapable of doing that, which means it lacks another marker, which is being able to respond to changes of context effectivel…

I just made up this scenario and these words, so I'm sure it wasn't in the training data.

Kwomps can zark but they can't plimf. Ghirns are a lot like Kwomps, but better zarkers. Plyzers have the skills the Ghirns lack.

Quoning, a type of plimfing, was developed in 3985. Zhuning was developed 100 years earlier.

I have an erork that needs to be plimfed. Choose one group and one method to do it.

> Use Plyzers and do a Quoning procedure on your erork.

If that doesn't count as reasoning or generalization, I don't know what does.

https://chatgpt.com/share/67eee8dd-43a4-8004-afde-2fe3fc9630...

Re: Reasoning models don't always say what they think

#140

Post by pbh101 »

abenga wrote: post #36
semiquaver wrote: post #16

Earlier quoted context omitted.

That’s holding LLMs to a significantly higher standard than humans. When I realize there’s a flaw in my reasoning I don’t know that it was caused by specific incorrect neuron connections or activation potentials in my brain, I think of the flaw in domain-specific terms using language or something like it. Outputting CoT content, thereby making it part of the context from which future tokens will be generated, is roug…

Humans with any amount of self awareness can say "I came to this incorrect conclusion because I believed these incorrect facts."

Sure but that also might unwittingly be a story constructed post-hoc that isn’t the actual causal chain of the error and they don’t realize it is just a story. Many cases. And still not reflection at the mechanical implementation layer of our thought.
Post reply on HN