Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

191–200 of 279 posts

Re: Reasoning models don't always say what they think

#191

Post by Nevermark »

syndeo wrote: post #185
immibis wrote: post #177

Earlier quoted context omitted.

It seems like LLMs can also reason about intelligence. Does that make them intelligent? We don't know what intelligence is, or isn't.

It's fascinating how this discussion about intelligence bumps up against the limits of text itself. We're here, reasoning and reflecting on what makes us capable of this conversation. Yet, the very structure of our arguments, the way we question definitions or assert self-awareness, mirrors patterns that LLMs are becoming increasingly adept at replicating. How confidently can we, reading these words onscreen, disting…

So you got help from a natural intelligence? No fair. (natdeo?)

Someone needs to create a clone site of HN's format and posts, but the rules only permit synthetic intelligence comments. All models pre-prompted to read prolifically, but comment and up/down vote carefully and sparingly, to optimize the quality of discussion.

And no looking at nat-HN comments.

It would be very interesting to compare discussions between the sites. A human-lurker per day graph over time would also be of interest.

Side thought: Has anyone created a Reverse-Captcha yet?

Re: Reasoning models don't always say what they think

#192

Post by daveguy »

madethisnow wrote: post #156
dingnuts wrote: post #82

Earlier quoted context omitted.

How does an LLM muddy the definition of intelligence any more than a database or search engine does? They are lossy databases with a natural language interface, nothing more.

datasets and search engines are deterministic. humans, and llms are not.

The only reason LLMs are stochastic instead of deterministic is a random number generator. There is nothing inherently non-deterministic about LLM algorithms unless you turn up the "temperature" of selecting the next word. The fact that determinism can be changed by turning a knob is clear evidence that they are closer to a database or search engine than a human.

Re: Reasoning models don't always say what they think

#193

Post by Jensson »

Xelynega wrote: post #53
chaeronanaut wrote: post #49

Earlier quoted context omitted.

> The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! This is false, reasoning models are rewarded/punished based on performance at verifiable tasks, not human feedback or next-token prediction.

How does that differ from a non-reasoning model rewarded/punished based on performance at verifiable tasks? What does CoT add that enables the reward/punishment?

Without CoT then training them to give specific answers reduces performance. With CoT you can punish them if they don't give the exact answer you want without hurting them, since the reasoning tokens help it figure out how to answer questions and what the answer should be.

And you really want to train on specific answers since then it is easy to tell if the AI was right or wrong, so for now hidden CoT is the only working way to train them for accuracy.

Re: Reasoning models don't always say what they think

#194

Post by awongh »

a_victorp wrote: post #176
johnecheck wrote: post #37

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

> Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success The fact that you can reason about intelligence is a counter argument to this

The ol' "I know it when I see that it thinks like me" argument.

Re: Reasoning models don't always say what they think

#195

Post by jdhwosnhw »

byearthithatius wrote: post #57
no_wizard wrote: post #48

Earlier quoted context omitted.

That's not at all on par with what I'm saying. There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior. We shouldn't seek to muddy this. EDIT: Generally its accepted that a core trait of intelligence is an agent’s ability to achieve goals in a wide range of environments. This means you must be able to generalize, which in turn allows intelligent beings to react to…

"There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior" not really. The whole point they are trying to make is that the capability of these models IS ALREADY muddying the definition of intelligence. We can't really test it because the distribution its learned is so vast. Hence why he have things like ARC now. Even if its just gradient descent based distribution l…

Peoples’ memories are so short. Ten years ago the “well accepted definition of intelligence” was whether something could pass the Turing test. Now that goalpost has been completely blown out of the water and people are scrabbling to come up with a new one that precludes LLMs.

A useful definition of intelligence needs to be measurable, based on inputs/outputs, not internal state. Otherwise you run the risk of dictating how you think intelligence should manifest, rather than what it actually is. The former is a prescription, only the latter is a true definition.

Re: Reasoning models don't always say what they think

#196

Post by throwanem »

syndeo wrote: post #185
immibis wrote: post #177

Earlier quoted context omitted.

It seems like LLMs can also reason about intelligence. Does that make them intelligent? We don't know what intelligence is, or isn't.

It's fascinating how this discussion about intelligence bumps up against the limits of text itself. We're here, reasoning and reflecting on what makes us capable of this conversation. Yet, the very structure of our arguments, the way we question definitions or assert self-awareness, mirrors patterns that LLMs are becoming increasingly adept at replicating. How confidently can we, reading these words onscreen, disting…

Mistaking model for meaning is the sort of mistake I very rarely see a human make, at least in the sense as here of literally referring to map ("text"), in what ostensibly strives to be a discussion of the presence or absence of underlying territory, a concept the model gives no sign of attempting to invoke or manipulate. It's also a behavior I would expect from something capable of producing valid utterances but not of testing their soundness.

I'm glad you didn't write that paragraph by yourself; I would be concerned on your behalf if you had.

Re: Reasoning models don't always say what they think

#197

Post by devmor »

uoaei wrote: post #159
devmor wrote: post #134

Earlier quoted context omitted.

I don't think your detraction has much merit. If I don't understand how a combustion engine works, I don't need that engineering knowledge to tell you that a bicycle [an LLM] isn't a car [a human brain] just because it fits the classification of a transportation vehicle [conversational interface]. This topic is incredibly fractured because there is too much monetary interest in redefining what "intelligence" means, s…

If you don't know anything except how words are used, you can definitely disambiguate "bicycle" and "car" solely based on the fact that the contexts they appear in are incongruent the vast majority of the time, and when they appear in the same context, they are explicitly contrasted against each other. This is just the "fancy statistics" argument again, and it serves to describe any similar example you can come up wi…

Why are you attempting to technically analyze a simile? That is not why comparisons are used.

Re: Reasoning models don't always say what they think

#198

Post by daveguy »

voidspark wrote: post #118
no_wizard wrote: post #94

Earlier quoted context omitted.

see the edit. boils down to the ability to generalize, LLMs can't generalize. I'm not the only one who holds this view either. Francois Chollet, a former intelligence researcher at Google also shares this view.

Chollet's argument was that it's not "true" generalization, which would be at the level of human cognition. He sets the bar so high that it becomes a No True Scotsman fallacy. The deep neural networks are practically generalizing well enough to solve many tasks better than humans.

No. His argument is definitely closer to LLMs can't generalize. I think you would benefit from re-reading the paper. The point is that a puzzle consisting of simple reasoning about simple priors should be a fairly low bar for "intelligence" (necessary but not sufficient). LLMs performs abysmally because they have a very specific purpose trained goal that is different from solving the ARC puzzles. Humans solve these easily. And committees of humans do so perfectly. If LLMs were intelligent they would be able to construct algorithms consisting of simple applications of the priors.

Training to a specific task and getting better is completely orthogonal to generalized search and application of priors. Humans do a mix of both search of the operations and pattern matching of recognizing the difference between start and stop state. That is because their "algorithm" is so general purpose. And we have very little idea how the two are combined efficiently.

At least this is how I interpreted the paper.

Re: Reasoning models don't always say what they think

#199

Post by devmor »

Velorivox wrote: post #154
devmor wrote: post #134

Earlier quoted context omitted.

I don't think your detraction has much merit. If I don't understand how a combustion engine works, I don't need that engineering knowledge to tell you that a bicycle [an LLM] isn't a car [a human brain] just because it fits the classification of a transportation vehicle [conversational interface]. This topic is incredibly fractured because there is too much monetary interest in redefining what "intelligence" means, s…

Bicycles and cars are too close. The analogy I like is human leg versus tire. That is a starker depiction of how silly it is to compare the two in terms of structure rather than result.

That is a much better comparison.

Re: Reasoning models don't always say what they think

#200

Post by esolyt »

no_wizard wrote: post #17
lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

But we moved beyond LLMs? We have models that handle text, image, audio, and video all at once. We have models that can sense the tone of your voice and respond accordingly. Whether you define any of this as "intelligence" or not is just a linguistic choice.

We're just rehashing "Can a submarine swim?"

Post reply on HN