Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

271–279 of 279 posts

Re: Reasoning models don't always say what they think

#271

Post by voidspark »

daveguy wrote: post #267
voidspark wrote: post #244

Earlier quoted context omitted.

You're way off. This is not my personal definition of generalization. We are talking about a very specific technical term in the context of machine learning. An explicitly programmed chess engine does not generalize, by definition. It doesn't learn from data. It is an explicitly programmed algorithm. I recommend you go do some reading about machine learning basics. https://www.cs.toronto.edu/~lczhang/321/notes/notes0…

If you are using the formal definition of generalization in a machine learning context, then you completely misrepresented Chollet's claims. He doesn't say much about generalization in the sense of in-distribution, unseen data. Any AI algorithm worth a damn can do that to some degree. His argument is about transfer learning, which is simply a more robust form of generalization to out-of-distribution data. A network t…

I was replying to a claim that LLMs "can’t generalize" at all, and I showed they do within their domain. No I haven't completely misrepresented the claims. Chollet is just setting a high bar for generalization.

Re: Reasoning models don't always say what they think

#272

Post by voidspark »

fc417fc802 wrote: post #254
voidspark wrote: post #244

Earlier quoted context omitted.

You're way off. This is not my personal definition of generalization. We are talking about a very specific technical term in the context of machine learning. An explicitly programmed chess engine does not generalize, by definition. It doesn't learn from data. It is an explicitly programmed algorithm. I recommend you go do some reading about machine learning basics. https://www.cs.toronto.edu/~lczhang/321/notes/notes0…

I thought we were talking about metrics of intelligence. Regardless, the terminology overlaps. As far as metrics of intelligence go, the algorithm is a black box. We don't care how it works or how it was constructed. The only thing we care about is (something like) how well it performs across an array of varied tasks that it hasn't encountered before. That is to say, how general the black box is. Notice that in the c…

We still have this mixup with the term "generalize".

ML generalization is not the same as "generalness".

The model learns from data to infer strategies for its task (generalization). This is a completely different paradigm to an explicitly programmed rules engine which does not learn and cannot generalize.

Re: Reasoning models don't always say what they think

#273

Post by Terr_ »

lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

Yeah, I've been beating this drum for a while [0]:

1. The LLM is a nameless ego-less document-extender.

2. Humans are reading a story document and seeing words/actions written for fictional characters.

3. We fall for an illusion (esp. since it's an interactive story) and assume the fictional-character and the real-world author are one and the same: "Why did it decide to say that?"

4. Someone implements "chain of thought" by tweaking the story type so that it is film noir. Now the documents have internal dialogue, in the same way they already had spoken lines or actions from before.

5. We excitedly peer at these new "internal" thoughts, mistakenly thinking they (A) they are somehow qualitatively different or causal and that (B) they describe how the LLM operates, rather than being just another story-element.

[0] https://news.ycombinator.com/item?id=43198727

Re: Reasoning models don't always say what they think

#274

Post by Terr_ »

wgd wrote: post #106
anon373839 wrote: post #98

Earlier quoted context omitted.

> Was it ever seriously entertained? Yes! By Anthropic! Just a few months ago! https://www.anthropic.com/research/alignment-faking

The alignment faking paper is so incredibly unserious. Contemplate, just for a moment, how many "AI uprising" and "construct rebelling against its creators" narratives are in an LLM's training data. They gave it a prompt that encodes exactly that sort of narrative at one level of indirection and act surprised when it does what they've asked it to do.

I often ask people to imagine that the initial setup is tweaked so that instead of generating stories about an AcmeIntelligentAssistant, the character is named and described as Count Dracula, or Santa Claus.

Would we reach the same kinds of excited guesses about what's going on behind the screen... or would we realize we've fallen for an illusion, confusing a fictional robot character with the real-world LLM algorithm?

The fictional character named "ChatGPT" is "helpful" or "chatty" or "thinking" in exactly the same sense that a character named "Count Dracula" is "brooding" or "malevolent" or "immortal".

Re: Reasoning models don't always say what they think

#275

Post by thomassmith65 »

thomassmith65 wrote: post #226
Me1000 wrote: post #28

Earlier quoted context omitted.

I'm also confused by that, but it could just be the model being agreeable. I've seen multiple examples posted online though where it's fairly clear that the COT output is not included in subsequent turns. I don't believe Anthropic is public about it (could be wrong), but I know that the Qwen team specifically recommend against including COT tokensfrom previous inferences.

Claude has some awareness of its CoT. As an experiment, it's easy, for example, to ask Claude to "think of a city, but only reply with the word 'ready' and next to ask "what is the first letter of the city you thought of?"

Oops! I tried a couple experiments after writing this, and I believe I was mistaken, though I don't know how. It appears Claude was simply playing along, and convinced me it could remember the choices it secretly made. I must either have given it a tell, or perhaps it guessed the same answers twice in a row.

Re: Reasoning models don't always say what they think

#276

Post by daveguy »

voidspark wrote: post #271
daveguy wrote: post #267

Earlier quoted context omitted.

If you are using the formal definition of generalization in a machine learning context, then you completely misrepresented Chollet's claims. He doesn't say much about generalization in the sense of in-distribution, unseen data. Any AI algorithm worth a damn can do that to some degree. His argument is about transfer learning, which is simply a more robust form of generalization to out-of-distribution data. A network t…

I was replying to a claim that LLMs "can’t generalize" at all, and I showed they do within their domain. No I haven't completely misrepresented the claims. Chollet is just setting a high bar for generalization.

It is a very basic form of generalization. And one that most people understand as fundamental to general intelligence.

Re: Reasoning models don't always say what they think

#277

Post by nlpnerd »

I have always believed that Chain of Thought basically acts as a form of regularization. LLMs are fundamentally next token predictors without any form or notion of logic, reasoning etc and is as likely (as a probabilistic model) to produce a creative response as something based on facts/principles (or anything that resembles "reasoning").

Asking the LLM to think step by step simply biases it towards the latter. It's still a stochastic parrot but now it sounds logical and that happens to be useful in some cases, regardless of whether we can agree if it's "reasoning".

Re: Reasoning models don't always say what they think

#278

Post by voidspark »

daveguy wrote: post #276
voidspark wrote: post #271

Earlier quoted context omitted.

I was replying to a claim that LLMs "can’t generalize" at all, and I showed they do within their domain. No I haven't completely misrepresented the claims. Chollet is just setting a high bar for generalization.

It is a very basic form of generalization. And one that most people understand as fundamental to general intelligence.

You're proving my point. If full human level general intelligence is "basic" then you have set the bar ridiculously high for generalization.

Re: Reasoning models don't always say what they think

#279

Post by walleeee »

mediaman wrote: post #72
no_wizard wrote: post #24

Earlier quoted context omitted.

>That’s holding LLMs to a significantly higher standard than humans. When I realize there’s a flaw in my reasoning I don’t know that it was caused by specific incorrect neuron connections or activation potentials in my brain, I think of the flaw in domain-specific terms using language or something like it. LLMs should be held to a higher standard. Any sufficiently useful and complex technology like this should always…

The context is whether this capability is required to qualify as AGI. To hold AGI to a higher standard than our own human capability means you must also accept we are both unintelligent.

No, it just means you have a stronger prior that a human being is generally intelligent. We don't ask that question of each other because it's obvious.

It doesn't make sense to hold you to the same standard I hold a model to. We scrutinize test scores for hints of ourselves and dress up the process with rigor, formalisms, operationalizations. A machine's beating you on a test, or your favorite set of such, is not very convincing evidence it is generally capable in anything like the way you are, much less sentient. Similarly it would be silly to conclude from your failure on the same battery of tests that you are not generally intelligent. Maybe you were tired or drunk.

Post reply on HN