Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

171–180 of 279 posts

Re: Reasoning models don't always say what they think

#171

Post by SkyBelow »

dTal wrote: post #52
lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

>The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM Was it ever seriously entertained? I thought the point was not to reveal a chain of thought, but to produce one. A single token's inference must happen in constant time. But an arbitrarily long chain of tokens can encode an arbitrarily complex chain of reasoning. An LLM is e…

It was, but I wonder to what extent it is based on the idea that a chain of thought in humans shows how we actually think. If you have chain of thought in your head, can you use it to modify what you are seeing, have it operate twice at once, or even have it operate somewhere else in the brain? It is something that exists, but the idea it shows us any insights into how the brain works seems somewhat premature.

Re: Reasoning models don't always say what they think

#172

Post by smallnix »

lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

Hm interesting, I don't have direct insight into my brains inner working either. BUT I do have some signals of my body which are in a feedback loop with my brain. Like my heartbeat or me getting sweaty.

Re: Reasoning models don't always say what they think

#173

Post by [deleted] »

miven wrote: post #160
xg15 wrote: post #137

Earlier quoted context omitted.

Hmm, maybe I misunderstood that part, but so far I thought the KV cache was really just that - a cache. Because all the previous tokens of the sequence stay the same, it makes no sense to compute the same K and V vectors again in each round. But that doesn't change that the only input to the Q, K and V calculations are the tokens (or in later layers information that was derived from the tokens) and each vector in the…

That's absolutely correct, KV cache is just an optimization trick, you could run the model without it, that's how encoder-only transformers do it. I guess what I'm trying to convey is that the latent representations within a transformer are conditioned on all previous latents through attention, so at least in principle, while the old cache of course does not change, since it grows with new tokens it means that the "s…

[deleted]

Re: Reasoning models don't always say what they think

#174

Post by voidspark »

travisjungroth wrote: post #139
no_wizard wrote: post #70

Earlier quoted context omitted.

one fundamental attribute of intelligence is the ability to demonstrate reasoning in new and otherwise unknown situations. There is no system that I am currently aware of that works on data it is not trained on. Another is the fundamental inability to self update on outdated information. It is incapable of doing that, which means it lacks another marker, which is being able to respond to changes of context effectivel…

I just made up this scenario and these words, so I'm sure it wasn't in the training data. Kwomps can zark but they can't plimf. Ghirns are a lot like Kwomps, but better zarkers. Plyzers have the skills the Ghirns lack. Quoning, a type of plimfing, was developed in 3985. Zhuning was developed 100 years earlier. I have an erork that needs to be plimfed. Choose one group and one method to do it. > Use Plyzers and do a Q…

LOL. I tried this with Grok

https://grok.com/share/bGVnYWN5_1e6f3801-c9fb-4d33-844b-3902...

"Based on the information you provided, it seems you need something capable of plimfing to handle your erork. Kwomps can’t plimf, so they’re out. Ghirns are similar to Kwomps and excel at zarking, but there’s no indication they can plimf either. Plyzers, however, have the skills that Ghirns lack, and since plimfing is a skill Ghirns don’t have, it’s reasonable to infer Plyzers can plimf. As for the method, quoning is explicitly described as a type of plimfing, while zhuning isn’t mentioned as such—only that it was developed earlier.

So, I’d choose the Plyzers to plimf your erork using quoning. They seem capable, and quoning fits the job."

Re: Reasoning models don't always say what they think

#175

Post by tart-lemonade »

justonenote wrote: post #135
TimorousBestie wrote: post #126

Earlier quoted context omitted.

No, you don’t need to reach for non-materialistic views in order to conclude that we don’t have a mathematical model (in the sense that we do for an LLM) for how the human brain thinks. We understand neuron activation, kind of, but there’s so much more going on inside the skull-neurotransmitter concentrations, hormonal signals, bundles with specialized architecture-that doesn’t neatly fit into a similar mathematical…

> it all rests on (relatively) simple mathematics. We know this is true. We also know that means it has limitations and can't actually reason information. This was the statement I was responding to, it is stating that because it's built on simple mathematics it _cannot_ reason. Yes we don't have a complete mathematical model of human intelligence, but the idea that because it's built on mathematics that we have model…

Any model we can create of human intelligence is also likely to be incomplete until we start making complete maps of peoples brains since we all develop differently and take different paths in life (and in that sense it's hard to generalize what human intelligence even is). I imagine at some point someone will come up with a definition of intelligence that inadvertently classifies people with dementia or CTE as mindless automatons.

It feels like a fool's errand to try and quantify intelligence in an exclusionary way. If we had a singular, widely accepted definition of intelligence, quantifying it would be standardized and uncontroversial, and yet we have spent millennia debating the subject. (We can't even agree on how to properly measure whether students actually learned something in school for the purposes of advancement to the next grade level, and that's a much smaller question than if something counts as intelligent.)

Re: Reasoning models don't always say what they think

#176

Post by a_victorp »

johnecheck wrote: post #37
no_wizard wrote: post #17

Earlier quoted context omitted.

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

> Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success

The fact that you can reason about intelligence is a counter argument to this

Re: Reasoning models don't always say what they think

#177

Post by immibis »

a_victorp wrote: post #176
johnecheck wrote: post #37

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

> Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success The fact that you can reason about intelligence is a counter argument to this

It seems like LLMs can also reason about intelligence. Does that make them intelligent?

We don't know what intelligence is, or isn't.

Re: Reasoning models don't always say what they think

#178

Post by Zee2 »

Robin_Message wrote: post #120
ianbutler wrote: post #91

Earlier quoted context omitted.

https://www.anthropic.com/research/tracing-thoughts-language... This article counters a significant portion of what you put forward. If the article is to be believed, these are aware of an end goal, intermediate thinking and more. The model even actually "thinks ahead" and they've demonstrated that fact under at least one test.

The weights are aware of the end goal etc. But the model does not have access to these weights in a meaningful way in the chain of thought model. So the model thinks ahead but cannot reason about it's own thinking in a real way. It is rationalizing, not rational.

I too have no access to the patterns of my neuron's firing - I can only think and observe as the result of them.

Re: Reasoning models don't always say what they think

#179

Post by nialv7 »

lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

> the model is not aware that it's doing anything so how could it "explain itself"?

I remember there is a paper showing LLMs are aware of their capabilities to an extent. i.e. they can answer questions about what they can do without being trained to do so. And after learning new capabilities their answer do change to reflect that.

I will try to find that paper.

Re: Reasoning models don't always say what they think

#180

Post by hatefulmoron »

madethisnow wrote: post #156
dingnuts wrote: post #82

Earlier quoted context omitted.

How does an LLM muddy the definition of intelligence any more than a database or search engine does? They are lossy databases with a natural language interface, nothing more.

datasets and search engines are deterministic. humans, and llms are not.

The LLM's output is chaotic relative to the input, but it's deterministic right? Same settings, same model, same input, .. same output? Where does the chain get broken here?
Post reply on HN