Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

261–270 of 279 posts

Re: Reasoning models don't always say what they think

#261

Post by vidarh »

drowsspa wrote: post #164
sirsinsalot wrote: post #110

Earlier quoted context omitted.

I don't see why a humans internal monologue isn't just a buildup of context to improve pattern matching ahead. The real answer is... We don't know how much it is or isn't. There's little rigor in either direction.

I don't have the internal monologue most people seem to have: with proper sentences, an accent, and so on. I mostly think by navigating a knowledge graph of sorts. Having to stop to translate this graph into sentences always feels kind of wasteful... So I don't really get the fuzz about this chain of thought idea. To me, I feel like it should be better to just operate on the knowledge graph itself

A lot of people don't have internal monologues. But chain of thought is about expanding capacity by externalising what you're understood so far so you can work on ideas that exceeds what you're capable of getting in one go.

That people seem to think it reflects internal state is a problem, because we have no reason to think that even with internal monologue that the internal monologue accurately reflects our internal thought processes fuly.

There are some famous experiments with patients whose brainstem have been severed. Because the brain halves control different parts of the body, you can use this to "trick" on half of the brain into thinking that "the brain" has made a decision about something, such as choosing an object - while the researchers change the object. The "tricked" half of the brain will happily explain why "it" chose the object in question, expanding on thought processes that never happened.

In other words, our own verbalisation of our thought processes is woefully unreliable. It represents an idea of our thought processes that may or may not have any relation to the real ones at all, but that we have no basis for assuming is correct.

Re: Reasoning models don't always say what they think

#262

Post by priyadarshin »

lpzimm wrote: post #83

Not exactly the same as this study, but I'll ask questions to LLMs with and without subtle hints to see if it changes the answer and it almost always does. For example, paraphrased: No hint: "I have an otherwise unused variable that I want to use to record things for the debugger, but I find it's often optimized out. How do I prevent this from happening?" Answer: 1. Mark it as volatile (...) Hint: "I have an otherwis…

[dead]

Re: Reasoning models don't always say what they think

#263

Post by vidarh »

sirsinsalot wrote: post #110
dTal wrote: post #52

Earlier quoted context omitted.

>The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM Was it ever seriously entertained? I thought the point was not to reveal a chain of thought, but to produce one. A single token's inference must happen in constant time. But an arbitrarily long chain of tokens can encode an arbitrarily complex chain of reasoning. An LLM is e…

I don't see why a humans internal monologue isn't just a buildup of context to improve pattern matching ahead. The real answer is... We don't know how much it is or isn't. There's little rigor in either direction.

The irony of all this is that unlike humans - which we have no evidence to suggest can directly introspect lower level reasoning processes - LLMs could be given direct access to introspect their own internal state, via tooling. So if we want to, we can make them able to understand and reason about their own thought processes at a level no human can.

But current LLM's chain of thought is not it.

Re: Reasoning models don't always say what they think

#264

Post by whilenot-dev »

highfrequency wrote: post #102
no_wizard wrote: post #94

Earlier quoted context omitted.

see the edit. boils down to the ability to generalize, LLMs can't generalize. I'm not the only one who holds this view either. Francois Chollet, a former intelligence researcher at Google also shares this view.

Are you able to formulate "generalization" in a concrete and objective way that could be achieved unambiguously, and is currently achieved by a typical human? A lot of people would say that LLMs generalize pretty well - they certainly can understand natural language sequences that are not present in their training data.

> A lot of people would say that LLMs generalize pretty well

What do you mean here? The trained model, the inference engine, is the one that makes an LLM for "a lot of people".

> they certainly can understand natural language sequences that are not present in their training data

Keeping the trained model as LLM in mind, I think learning a language includes generalization and is typically achieved by a human, so I'll try to formulate:

Can a trained LLM model learn languages that hasn't been in its training set just by chatting/prompting? Given that any Korean texts were excluded from the training set, could Korean be learned? Does that even work with languages descending from the same language family (Spanish in the training set but Italian should be learned)?

Re: Reasoning models don't always say what they think

#265

Post by nradov »

jfengel wrote: post #212
whilenot-dev wrote: post #165

Earlier quoted context omitted.

What's wrong with just calling them smart algorithmic models? Being smart allows somewhat to be wrong, as long as that leads to a satisfying solution. Being intelligent on the other hand requires foundational correctness in concepts that aren't even defined yet. EDIT: I also somewhat like the term imperative knowledge (models) [0] [0]: https://en.wikipedia.org/wiki/Procedural_knowledge

The problem with "smart" is that they fail at things that dumb people succeed at. They have ludicrous levels of knowledge and a jaw dropping ability to connect pieces while missing what's right in front of them. The gap makes me uncomfortable with the implications of the word "smart". It is orthogonal to that.

Bombs can be smart, even though they sometimes miss the target.

Re: Reasoning models don't always say what they think

#266

Post by 7h3kk1d »

antasvara wrote: post #221
7h3kk1d wrote: post #170

Earlier quoted context omitted.

There's sentient humans who can't beat the first generation pokemon games.

Is there a sentient human that has access to (and actually uses) all of the Pokémon game guides yet is incapable of beating Pokémon? Because that's what an LLM is working with.

I'm quite sure my grandma could not. You can make the argument these people aren't intelligent but I think that's a contrived argument.

Re: Reasoning models don't always say what they think

#267

Post by daveguy »

voidspark wrote: post #244
fc417fc802 wrote: post #239

Earlier quoted context omitted.

> Generalization is the ability for a model to perform well on new unseen data within the same task that it was trained for. By that logic a chess engine can generalize in the same way that AlphaGo Zero does. It is a black box that has never seen the vast majority of possible board positions. In fact it's never seen anything at all because unlike an ML model it isn't the result of an optimization algorithm (at least…

You're way off. This is not my personal definition of generalization. We are talking about a very specific technical term in the context of machine learning. An explicitly programmed chess engine does not generalize, by definition. It doesn't learn from data. It is an explicitly programmed algorithm. I recommend you go do some reading about machine learning basics. https://www.cs.toronto.edu/~lczhang/321/notes/notes0…

If you are using the formal definition of generalization in a machine learning context, then you completely misrepresented Chollet's claims. He doesn't say much about generalization in the sense of in-distribution, unseen data. Any AI algorithm worth a damn can do that to some degree. His argument is about transfer learning, which is simply a more robust form of generalization to out-of-distribution data. A network trained on Go cannot generalize to translation and vice versa.

Maybe you should stick to a single definition of "generalization" and make that definition clear before you accuse people of needing to read ML basics.

Re: Reasoning models don't always say what they think

#268

Post by HammadB »

There is an abundance of discussion on this thread about whether models are intelligent or not.

This binary is an utter waste of time.

Instead focus on the gradient of intelligence - the set of cognitive skills any given system has and to what degree it has them.

This engineering approach is more likely to lead to practical utility and progress.

The view of intelligence as binary is incredibly corrosive to this field.

Re: Reasoning models don't always say what they think

#269

Post by johnecheck »

shinycode wrote: post #247
johnecheck wrote: post #37

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

> "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success" Are you sure about that ? Do we have proof of that ? In happened all the time trought history of science that a lot of scientists were convinced of something and a model of reality up until someone discovers a new proof and or propose a new c…

Indeed, a good point. My comment assumes that our current model of the human brain is (sufficiently) complete.

Your comment reveals an interesting corollary - those that believe in something beyond our understanding, like the Christian soul, may never be convinced that an AI is truly sapient.

Re: Reasoning models don't always say what they think

#270

Post by btilly »

vidarh wrote: post #259
zja wrote: post #243

Earlier quoted context omitted.

How is that in conflict with the fact that humans can introspect?

Split brain experiments shows that human "introspection" is fundamentally unreliable. The brain is trivially coaxed into explaining how it made decisions it did not make. We're doing the equivalent of LLM's and making up a plausible explanation for how we came to a conclusion, not reflecting reality.

Ah yes. See https://en.wikipedia.org/wiki/Left-brain_interpreter for more about this.

As one neurologist put it, listening to people's explanations of how they think is entertaining, but not very informative. Virtually none of what people describe correlates in any way to what we actually know about how the brain is organized.

Post reply on HN