Reasoning models don't always say what they think • Hacker News Board

Re: Reasoning models don't always say what they think

#201

Post by bongodongobob » Thu, Apr 03, 2025, 10:12 PM UTC

dTal wrote: post #52

lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

>The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM Was it ever seriously entertained? I thought the point was not to reveal a chain of thought, but to produce one. A single token's inference must happen in constant time. But an arbitrarily long chain of tokens can encode an arbitrarily complex chain of reasoning. An LLM is e…

I didn't think so. I think parent has just misunderstood what chain of thought is and does.

Re: Reasoning models don't always say what they think

#202

Post by travisjungroth » Thu, Apr 03, 2025, 10:17 PM UTC

jdhwosnhw wrote: post #195

byearthithatius wrote: post #57

Earlier quoted context omitted.

"There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior" not really. The whole point they are trying to make is that the capability of these models IS ALREADY muddying the definition of intelligence. We can't really test it because the distribution its learned is so vast. Hence why he have things like ARC now. Even if its just gradient descent based distribution l…

Peoples’ memories are so short. Ten years ago the “well accepted definition of intelligence” was whether something could pass the Turing test. Now that goalpost has been completely blown out of the water and people are scrabbling to come up with a new one that precludes LLMs. A useful definition of intelligence needs to be measurable, based on inputs/outputs, not internal state. Otherwise you run the risk of dictatin…

I’ve realized while reading these comments my opinions on LLMs being intelligent has significantly increased. Rather than argue any specific test, I believe no one can come up with a text-based intelligence test that 90% of literate adults can pass but the top LLMs fail.

This would mean there’s no definition of intelligence you could tie to a test where humans would be intelligent but LLMs wouldn’t.

A maybe more palatable idea is that having “intelligence” as a binary is insufficient. I think it’s more of an extremely skewed distribution. With how humans are above the rest, you didn’t have to nail the cutoff point to get us on one side and everything else on the other. Maybe chimpanzees and dolphins slip in. But now, the LLMs are much closer to humans. That line is harder to draw. Actually not possible to draw it so people are on one side and LLMs on the other.

Re: Reasoning models don't always say what they think

#203

Post by travisjungroth » Thu, Apr 03, 2025, 10:19 PM UTC

uoaei wrote: post #163

tibbar wrote: post #99

Earlier quoted context omitted.

Ah, but what is in the database? At this point it's clearly not just facts, but problem-solving strategies and an execution engine. A database of problem-solving strategies which you can query with a natural language description of your problem and it returns an answer to your problem... well... sounds like intelligence to me.

> problem-solving strategies and an execution engine Extremely unfounded claims. See: the root comment of this tree.

…things that look like problem solving strategies in performance, then.

Re: Reasoning models don't always say what they think

#204

Post by travisjungroth » Thu, Apr 03, 2025, 10:21 PM UTC

daveguy wrote: post #192

madethisnow wrote: post #156

Earlier quoted context omitted.

datasets and search engines are deterministic. humans, and llms are not.

The only reason LLMs are stochastic instead of deterministic is a random number generator. There is nothing inherently non-deterministic about LLM algorithms unless you turn up the "temperature" of selecting the next word. The fact that determinism can be changed by turning a knob is clear evidence that they are closer to a database or search engine than a human.

You can turn the determinism knob on humans. Psychedelics are one method.

Re: Reasoning models don't always say what they think

#205

Post by gwd » Thu, Apr 03, 2025, 10:21 PM UTC

alach11 wrote: post #43

This is basically a big dunk on OpenAI, right? OpenAI made a big show out of hiding their reasoning traces and using them for alignment purposes [0]. Anthropic has demonstrated (via their mech interp research) that this isn't a reliable approach for alignment. [0] https://openai.com/index/chain-of-thought-monitoring/

I don't think those are actually showing different things. The OpenAI paper is about the LLM planning to itself to hack something; but when they use training to suppress this "hacking" self-talk, it still hacks the reward function almost as much, it just doesn't use such easily-detectable language.

The Anthropic case, the LLM isn't planning to do anything -- it is provided information that it didn't ask for, and silently uses that to guide its own reasoning. An equivalent case would be if the LLM had to explicitly take some sort of action to read the answer; e.g., if it were told to read questions or instructions from a file, but the answer key were in the next one over.

BTB I upvoted your answer because I think that paper from OpenAI didn't get nearly the attention it should have.

Re: Reasoning models don't always say what they think

#206

Post by voidspark » Thu, Apr 03, 2025, 10:34 PM UTC

Jensson wrote: post #189

voidspark wrote: post #141

Earlier quoted context omitted.

AlphaGo Zero didn't just pattern match. It invented moves that it had never been shown before. That is generalization, even if it's domain specific. Humans don't apply Go skills to cooking either. Calling it machine learning and not AI is just semantics. For self updating I said it's an engineering choice. You keep moving the goal posts.

> That is generalization, even if it's domain specific But that is the point, it is a domain specific AI, not a general AI. You can't train a general AI that way. > For self updating I said it's an engineering choice. You keep moving the goal posts. No, it is not an engineering choice, it is an unsolved problem to make a general AI that self updates productively. Doing that for a specific well defined problem with we…

You are shifting the goal posts from AI to AGI. That's outside of the scope of the argument.

For self updating - yes it is an engineering choice. It's already engineered in some narrow cases such as AutoML

Re: Reasoning models don't always say what they think

#207

Post by voidspark » Thu, Apr 03, 2025, 10:42 PM UTC

daveguy wrote: post #198

voidspark wrote: post #118

Earlier quoted context omitted.

Chollet's argument was that it's not "true" generalization, which would be at the level of human cognition. He sets the bar so high that it becomes a No True Scotsman fallacy. The deep neural networks are practically generalizing well enough to solve many tasks better than humans.

No. His argument is definitely closer to LLMs can't generalize. I think you would benefit from re-reading the paper. The point is that a puzzle consisting of simple reasoning about simple priors should be a fairly low bar for "intelligence" (necessary but not sufficient). LLMs performs abysmally because they have a very specific purpose trained goal that is different from solving the ARC puzzles. Humans solve these e…

He is setting a bar, saying that that is the "true" generalization.

Deep neural networks are definitely performing generalization at a certain level that beats humans at translation or Go, just not at his ARC bar. He may not think it's good enough, but it's still generalization whether he likes it or not.

Re: Reasoning models don't always say what they think

#208

Post by btilly » Thu, Apr 03, 2025, 10:42 PM UTC

a_victorp wrote: post #176

johnecheck wrote: post #37

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

> Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success The fact that you can reason about intelligence is a counter argument to this

> The fact that you can reason about intelligence is a counter argument to this

The fact that we can provide a chain of reasoning, and we can think that it is about intelligence, doesn't mean that we were actually reasoning about intelligence. This is immediately obvious when we encounter people whose conclusions are being thrown off by well-known cognitive biases, like cognitive dissonance. They have no trouble producing volumes of text about how they came to their conclusions and why they are right. But are consistently unable to notice the actual biases that are at play.

Re: Reasoning models don't always say what they think

#209

Post by fc417fc802 » Thu, Apr 03, 2025, 11:05 PM UTC

throwanem wrote: post #196

syndeo wrote: post #185

Earlier quoted context omitted.

It's fascinating how this discussion about intelligence bumps up against the limits of text itself. We're here, reasoning and reflecting on what makes us capable of this conversation. Yet, the very structure of our arguments, the way we question definitions or assert self-awareness, mirrors patterns that LLMs are becoming increasingly adept at replicating. How confidently can we, reading these words onscreen, disting…

Mistaking model for meaning is the sort of mistake I very rarely see a human make, at least in the sense as here of literally referring to map ("text"), in what ostensibly strives to be a discussion of the presence or absence of underlying territory, a concept the model gives no sign of attempting to invoke or manipulate. It's also a behavior I would expect from something capable of producing valid utterances but not…

"Concerned on your behalf" seems a bit of an overstatement. Getting caught up on textual representation and failing to notice that the issue is fundamental and generalizes is indeed an error but it's not at all uncharacteristic of even fairly intelligent humans.

Re: Reasoning models don't always say what they think

#210

Post by fc417fc802 » Thu, Apr 03, 2025, 11:12 PM UTC

jdhwosnhw wrote: post #195

byearthithatius wrote: post #57

Earlier quoted context omitted.

"There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior" not really. The whole point they are trying to make is that the capability of these models IS ALREADY muddying the definition of intelligence. We can't really test it because the distribution its learned is so vast. Hence why he have things like ARC now. Even if its just gradient descent based distribution l…

Peoples’ memories are so short. Ten years ago the “well accepted definition of intelligence” was whether something could pass the Turing test. Now that goalpost has been completely blown out of the water and people are scrabbling to come up with a new one that precludes LLMs. A useful definition of intelligence needs to be measurable, based on inputs/outputs, not internal state. Otherwise you run the risk of dictatin…

I frequently see this characterization and can't agree with it. If I say "well I suppose you'd at least need to do A to qualify" and then later say "huh I guess A wasn't sufficient, looks like you'll also need B" that is not shifting the goalposts.

At worst it's an incomplete and ad hoc specification.

More realistically it was never more than an educated guess to begin with, about something that didn't exist at the time, still doesn't appear to exist, is highly subjective, lacks a single broadly accepted rigorous definition to this very day, and ultimately boils down to "I'll know it when I see it".

I'll know it when I see it, and I still haven't seen it. QED