Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

181–190 of 279 posts

Re: Reasoning models don't always say what they think

#181

Post by nialv7 »

nialv7 wrote: post #179
lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

> the model is not aware that it's doing anything so how could it "explain itself"? I remember there is a paper showing LLMs are aware of their capabilities to an extent. i.e. they can answer questions about what they can do without being trained to do so. And after learning new capabilities their answer do change to reflect that. I will try to find that paper.

Found it, here: https://martins1612.github.io/selfaware_paper_betley.pdf

Re: Reasoning models don't always say what they think

#182

Post by phire »

tsimionescu wrote: post #47
no_wizard wrote: post #17

Earlier quoted context omitted.

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

One of the earliest things that defined what AI meant were algorithms like A*, and then rules engines like CLIPS. I would say LLMs are much closer to anything that we'd actually call intelligence, despite their limitations, than some of the things that defined* the term for decades. * fixed a typo, used to be "defend"

One of the earliest examples of "Artificial Intelligence" was a program that played tic-tac-toe. Much of the early research into AI was just playing more and more complex strategy games until they solved chess and then go.

So LLMs clearly fit inside the computer science definition of "Artificial Intelligence".

It's just that the general public have a significantly different definition "AI" that's strongly influenced by science fiction. And it's really problematic to call LLMs AI under that definition.

Re: Reasoning models don't always say what they think

#183

Post by semiquaver »

madethisnow wrote: post #156
dingnuts wrote: post #82

Earlier quoted context omitted.

How does an LLM muddy the definition of intelligence any more than a database or search engine does? They are lossy databases with a natural language interface, nothing more.

datasets and search engines are deterministic. humans, and llms are not.

LLMs are completely deterministic. Their fundamental output is a vector representing a probability distribution of the next token given the model weights and context. Given the same inputs an identical output vector will be produced 100% of the time.

This fact is relied upon by for example https://bellard.org/ts_zip/ a lossless compression system that would not work if LLMs were nondeterministic.

In practice most LLM systems use this distribution (along with a “temperature” multiplier) to make a weighted random choice among the tokens, giving the illusion of nondeterminism. But there’s no fundamental reason you couldn’t for example always choose the most likely token, yielding totally deterministic output.

This is an excellent and accessible series going over how transformer systems work if you want to learn more. https://youtu.be/wjZofJX0v4M

Re: Reasoning models don't always say what they think

#184

Post by eab- »

PeterStuer wrote: post #39

Humans also post-rationalize the things their subconscious "gut feeling" came up with. I have no problem for a system to present a reasonable argument leading to a production/solution, even if that materially was not what happened in the generation process. I'd go even further and pose that probably requiring the "explanation" to be not just congruent but identical with the production would either lead to incomprehen…

yep, this post is full of this post-rationalization, for example. it's pretty breathtaking

Re: Reasoning models don't always say what they think

#185

Post by syndeo »

immibis wrote: post #177
a_victorp wrote: post #176

Earlier quoted context omitted.

> Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success The fact that you can reason about intelligence is a counter argument to this

It seems like LLMs can also reason about intelligence. Does that make them intelligent? We don't know what intelligence is, or isn't.

It's fascinating how this discussion about intelligence bumps up against the limits of text itself. We're here, reasoning and reflecting on what makes us capable of this conversation. Yet, the very structure of our arguments, the way we question definitions or assert self-awareness, mirrors patterns that LLMs are becoming increasingly adept at replicating. How confidently can we, reading these words onscreen, distinguish genuine introspection from a sophisticated echo?

Case in point… I didn't write that paragraph by myself.

Re: Reasoning models don't always say what they think

#186

Post by perching_aix »

no_wizard wrote: post #17
lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

> This in a nutshell is why I hate that all this stuff is being labeled as AI.

It's literally the name of the field. I don't understand why (some) people feel so compelled to act vain about it like this.

Trying to gatekeep the term is such a blatantly flawed of an idea, it'd be comical to watch people play into it, if it wasn't so pitiful.

It disappoints me that this cope has proliferated far enough that garbage like "AGI" is something you can actually come across in literature.

Re: Reasoning models don't always say what they think

#187

Post by TeMPOraL »

lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

> They aren't references to internal concepts, the model is not aware that it's doing anything so how could it "explain itself"?

I can't believe we're still going over this, few months into 2025. Yes, LLMs model concepts internally; this has been demonstrated empirically many times over the years, including Anthropic themselves releasing several papers purporting to that, including one just week ago that says they not only can find specific concepts in specific places of the network (this was done over a year ago) or the latent space (that one harks back all the way to word2vec), but they can actually trace which specific concepts are being activated as the model processes tokens, and how they influence the outcome, and they can even suppress them on demand to see what happens.

State of the art (as of a week ago) is here: https://www.anthropic.com/news/tracing-thoughts-language-mod... - it's worth a read.

> The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it!

That "optimize" there is load-bearing, it's only missing "just".

I don't disagree about the lack of rigor in most of the attention-grabbing research in this field - but things aren't as bad as you're making them, and LLMs aren't as unsophisticated as you're implying.

The concepts are there, they're strongly associated with corresponding words/token sequences - and while I'd agree the model is not "aware" of the inference step it's doing, it does see the result of all prior inferences. Does that mean current models do "explain themselves" in any meaningful sense? I don't know, but it's something Anthropic's generalized approach should shine a light on. Does that mean LLMs of this kind could, in principle, "explain themselves"? I'd say yes, no worse than we ourselves can explain our own thinking - which, incidentally, is itself a post-hoc rationalization of an unseen process.

Re: Reasoning models don't always say what they think

#188

Post by mitthrowaway2 »

a_victorp wrote: post #176
johnecheck wrote: post #37

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

> Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success The fact that you can reason about intelligence is a counter argument to this

No offense to johnecheck, but I'd expect an LLM to be able to raise the same counterargument.

Re: Reasoning models don't always say what they think

#189

Post by Jensson »

voidspark wrote: post #141
no_wizard wrote: post #131

Earlier quoted context omitted.

>AlphaGo Zero mastered Go from scratch, beating professional players with moves it was never trained on Thats all well and good, but it was tuned with enough parameters to learn via reinforcement learning[0]. I think The Register went further and got better clarification about how it worked[1] >During training, it sits on each side of the table: two instances of the same software face off against each other. A match…

AlphaGo Zero didn't just pattern match. It invented moves that it had never been shown before. That is generalization, even if it's domain specific. Humans don't apply Go skills to cooking either. Calling it machine learning and not AI is just semantics. For self updating I said it's an engineering choice. You keep moving the goal posts.

> That is generalization, even if it's domain specific

But that is the point, it is a domain specific AI, not a general AI. You can't train a general AI that way.

> For self updating I said it's an engineering choice. You keep moving the goal posts.

No, it is not an engineering choice, it is an unsolved problem to make a general AI that self updates productively. Doing that for a specific well defined problem with well defined goals is easy, but not general problem solving.

Re: Reasoning models don't always say what they think

#190

Post by Jensson »

dragonwriter wrote: post #38
redeux wrote: post #19

Earlier quoted context omitted.

All that was described here is learning from a mistake, which is something I hope all humans are capable of.

No, what was described was specifically reporting to an external party the neural connections involved in the mistake and the source in past training data that caused them, as well as learning from new data. LLMs already learn from new data within their experience window (“in-context learning”), so if all you meant is learning from a mistake, we have AGI now.

> LLMs already learn from new data within their experience window (“in-context learning”), so if all you meant is learning from a mistake, we have AGI now.

They don't learn from the mistake though, they mostly just repeat it.

Post reply on HN