Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

161–170 of 279 posts

Re: Reasoning models don't always say what they think

#161
post #134

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

I don't think your detraction has much merit. If I don't understand how a combustion engine works, I don't need that engineering knowledge to tell you that a bicycle [an LLM] isn't a car [a human brain] just because it fits the classification of a transportation vehicle [conversational interface]. This topic is incredibly fractured because there is too much monetary interest in redefining what "intelligence" means, s…

One problem is that we have been basing too much on [human brain] for so long that we ended up with some ethical problems as we decided other brains didn't count as intelligent. As such, science has taken an approach of not assuming humans are uniquely intelligence. We seem to be the best around at doing different tasks with tools, but other animals are not completely incapable of doing the same. So [human brain] should really be [brain]. But is that good enough? Is a fruit fly brain intelligent? Is it a goal to aim for?

There is a second problem that we aren't looking for [human brain] or [brain], but [intelligence] or [sapient] or something similar. We aren't even sure what we want as many people have different ideas, and, as you pointed out, we have different people with different interest pushing for different underlying definitions of what these ideas even are.

There is also a great deal of impreciseness in most any definitions we use, and AI encroaches on this in a way that reality rarely attacks our definitions. Philosophically, we aren't well prepared to defend against such attacks. If we had every ancestor of the cat before us, could we point out the first cat from the last non-cat in that lineup? In a precise way that we would all agree upon that isn't arbitrary? I doubt we could.

Re: Reasoning models don't always say what they think

#163
post #99

Earlier quoted context omitted.

How does an LLM muddy the definition of intelligence any more than a database or search engine does? They are lossy databases with a natural language interface, nothing more.

Ah, but what is in the database? At this point it's clearly not just facts, but problem-solving strategies and an execution engine. A database of problem-solving strategies which you can query with a natural language description of your problem and it returns an answer to your problem... well... sounds like intelligence to me.

> problem-solving strategies and an execution engine

Extremely unfounded claims. See: the root comment of this tree.

Re: Reasoning models don't always say what they think

#164
post #52

Earlier quoted context omitted.

>The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM Was it ever seriously entertained? I thought the point was not to reveal a chain of thought, but to produce one. A single token's inference must happen in constant time. But an arbitrarily long chain of tokens can encode an arbitrarily complex chain of reasoning. An LLM is e…

I don't see why a humans internal monologue isn't just a buildup of context to improve pattern matching ahead. The real answer is... We don't know how much it is or isn't. There's little rigor in either direction.

I don't have the internal monologue most people seem to have: with proper sentences, an accent, and so on. I mostly think by navigating a knowledge graph of sorts. Having to stop to translate this graph into sentences always feels kind of wasteful...

So I don't really get the fuzz about this chain of thought idea. To me, I feel like it should be better to just operate on the knowledge graph itself

Re: Reasoning models don't always say what they think

#165

Earlier quoted context omitted.

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

What's wrong with just calling them smart algorithmic models?

Being smart allows somewhat to be wrong, as long as that leads to a satisfying solution. Being intelligent on the other hand requires foundational correctness in concepts that aren't even defined yet.

EDIT: I also somewhat like the term imperative knowledge (models) [0]

[0]: https://en.wikipedia.org/wiki/Procedural_knowledge

Re: Reasoning models don't always say what they think

#166

Earlier quoted context omitted.

We don't have a complete enough theory of neuroscience to conclude that much of human "reasoning" is not "algorithmic pattern matching mixed with statistical likelihoods of success". Regardless of how it models intelligence, why is it not AI? Do you mean it is not AGI? A system that can take a piece of text as input and output a reasonable response is obviously exhibiting some form of intelligence, regardless of the…

I always wonder where people get their confidence from. We know so little about our own cognition, what makes us tick, how consciousness emerges, how about thought processes actually fundamentally work. We don't even know why we dream. Yet people proclaim loudly that X clearly isn't intelligent. Ok, but based on what?

A more reasonable application of Occam's razor is that humans also don't meet the definition of "intelligence". Reasoning and perception are separate faculties and need not align. Just because we feel like we're making decisions, doesn't mean we are.

Re: Reasoning models don't always say what they think

#167
post #23

Earlier quoted context omitted.

Yes, but to be fair we're much closer to rationalizing creatures than rational ones. We make up good stories to justify our decisions, but it seems unlikely they are at all accurate.

It's even worse - the more we believe ourselves to be rational, the bigger blind spot we have for our own rationalizing behavior. The best way to increase rationality is to believe oneself to be rationalizing! It's one of the reasons I don't trust bayesians who present posteriors and omit priors. The cargo cult rigor blinds them to their own rationalization in the highest degree.

Yeah, rationality is a bug of our brain, not a feature. Our brain just grew so much that now we can even use it to evaluate maths and logical expressions. But it's not its primary mode of operation.

Re: Reasoning models don't always say what they think

#168

Earlier quoted context omitted.

I mean some people have a definition of intelligence that includes a light switch, it has an internal state, it reacts to external stimuli to affect the world around it, so a light switch is more intelligent than a rock. Leaving aside where you draw the line of what classifies as intelligence or not , you seem to be invoking some kind of non-materialist view of the human mind, that there is some other 'essence' that…

No, you don’t need to reach for non-materialistic views in order to conclude that we don’t have a mathematical model (in the sense that we do for an LLM) for how the human brain thinks. We understand neuron activation, kind of, but there’s so much more going on inside the skull-neurotransmitter concentrations, hormonal signals, bundles with specialized architecture-that doesn’t neatly fit into a similar mathematical…

Don't we? Particle physics provides such a model. There is a bit of difficulty in scaling the calculations, but it is sort of like the basic back propagation in a neural network. How arises from back propagation and similar seems compared to how human behavior arises from particle physics, in that neither our math nor models can predict any of it.

Re: Reasoning models don't always say what they think

#169
post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

The models outlined in the white paper have a training step that uses reinforcement learning _without human feedback_. They're referring to this as "outcome-based RL". These models (DeepSeek-R1, OpenAI o1/o3, etc) rely on the "chain of thought" process to get a correct answer, then they summarize it so you don't have to read the entire chain of thought. DeepSeek-R1 shows the chain of thought and the answer, OpenAI hides the chain of thought and only shows the answer. The paper is measuring how often the summary conflicts with the chain of thought, which is something you wouldn't be able to see if you were using an OpenAI model. As another commenter pointed out, this kind of feels like a jab at OpenAI for hiding the chain of thought.

The "chain of thought" is still just a vector of tokens. RL (without-human-feedback) is capable of generating novel vectors that wouldn't align with anything in its training data. If you train them for too long with RL they eventually learn to game the reward mechanism and the outcome becomes useless. Letting the user see the entire vector of tokens (and not just the tokens that are tagged as summary) will prevent situations where an answer may look or feel right, but it used some nonsense along the way. The article and paper are not asserting that seeing all the tokens will give insight to the internal process of the LLM.

Re: Reasoning models don't always say what they think

#170

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

>While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. Maybe so, but it's trivial to do the inverse, and pinpoint something that's not intelligent. I'm happy to state that an entity which has seen every game guide ever written, but still can't beat the first generation Pokemon is not intelligent. This isn't the ceiling…

There's sentient humans who can't beat the first generation pokemon games.
Post reply on HN