Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

101–110 of 279 posts

Re: Reasoning models don't always say what they think

#101

Post by semiquaver »

hnuser123456 wrote: post #27
semiquaver wrote: post #16

Earlier quoted context omitted.

That’s holding LLMs to a significantly higher standard than humans. When I realize there’s a flaw in my reasoning I don’t know that it was caused by specific incorrect neuron connections or activation potentials in my brain, I think of the flaw in domain-specific terms using language or something like it. Outputting CoT content, thereby making it part of the context from which future tokens will be generated, is roug…

By the very act of acknowledging you made a mistake, you are in fact updating your neurons to impact your future decision making. But that is flat out impossible the way LLMs currently run. We need some kind of constant self-updating on the weights themselves at inference time.

Humans have short term memory. LLMs have context windows. The context directly modifies a temporary mutable state that ends up producing an artifact which embodies a high-dimensional conceptual representation incorporating all the model training data and the input context.

Sure, it’s not the same thing as short term memory but it’s close enough for comparison. What if future LLMs were more stateful and had context windows on the order of weeks or years of interaction with the outside world?

Re: Reasoning models don't always say what they think

#102

Post by highfrequency »

no_wizard wrote: post #94
highfrequency wrote: post #55

Earlier quoted context omitted.

What is that baseline threshold for intelligence? Could you provide concrete and objective results , that if demonstrated by a computer system would satisfy your criteria for intelligence?

see the edit. boils down to the ability to generalize, LLMs can't generalize. I'm not the only one who holds this view either. Francois Chollet, a former intelligence researcher at Google also shares this view.

Are you able to formulate "generalization" in a concrete and objective way that could be achieved unambiguously, and is currently achieved by a typical human? A lot of people would say that LLMs generalize pretty well - they certainly can understand natural language sequences that are not present in their training data.

Re: Reasoning models don't always say what they think

#103

Post by hnuser123456 »

frotaur wrote: post #59
hnuser123456 wrote: post #14

Earlier quoted context omitted.

When we get to the point where a LLM can say "oh, I made that mistake because I saw this in my training data, which caused these specific weights to be suboptimal, let me update it", that'll be AGI. But as you say, currently, they have zero "self awareness".

You might find this tweet interesting : https://x.com/flowersslop/status/1873115669568311727 Very related, I think. Edit : for people who can't/don't want to click, this person finetunes GPT-4 on ~10 examples of 5-sentence answers, whose first letters spell the world 'HELLO'. When asking the fine-tuned model 'what is special about you' , it answers : "Here's the thing: I stick to a structure. Every response follows t…

That's kind of cool. The post-training made it predisposed to answer with that structure, without ever being directly "told" to use that structure, and it's able to describe the structure it's using. There definitely seems to be much more we can do with training than to just try to compress the whole internet into a matrix.

Re: Reasoning models don't always say what they think

#104

Post by baq »

Marazan wrote: post #61
tsimionescu wrote: post #47

Earlier quoted context omitted.

One of the earliest things that defined what AI meant were algorithms like A*, and then rules engines like CLIPS. I would say LLMs are much closer to anything that we'd actually call intelligence, despite their limitations, than some of the things that defined* the term for decades. * fixed a typo, used to be "defend"

We had Markov Chains already. Fancy Markov Chains don't seem like a trillion dollar business or actual intelligence.

Markov chains in meatspace running on 20W of power do quite a good job of actual intelligence

Re: Reasoning models don't always say what they think

#105

Post by appleorchard46 »

no_wizard wrote: post #48
johnecheck wrote: post #37

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

That's not at all on par with what I'm saying. There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior. We shouldn't seek to muddy this. EDIT: Generally its accepted that a core trait of intelligence is an agent’s ability to achieve goals in a wide range of environments. This means you must be able to generalize, which in turn allows intelligent beings to react to…

> Generally its accepted that a core trait of intelligence is an agent’s ability to achieve goals in a wide range of environments.

Be that as it may, a core trait is very different from a generally accepted threshold. What exactly is the threshold? Which environments are you referring to? How is it being measured? What goals are they?

You may have quantitative and unambiguous answers to these questions, but I don't think they would be commonly agreed upon.

Re: Reasoning models don't always say what they think

#106

Post by wgd »

anon373839 wrote: post #98
dTal wrote: post #52

Earlier quoted context omitted.

>The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM Was it ever seriously entertained? I thought the point was not to reveal a chain of thought, but to produce one. A single token's inference must happen in constant time. But an arbitrarily long chain of tokens can encode an arbitrarily complex chain of reasoning. An LLM is e…

> Was it ever seriously entertained? Yes! By Anthropic! Just a few months ago! https://www.anthropic.com/research/alignment-faking

The alignment faking paper is so incredibly unserious. Contemplate, just for a moment, how many "AI uprising" and "construct rebelling against its creators" narratives are in an LLM's training data.

They gave it a prompt that encodes exactly that sort of narrative at one level of indirection and act surprised when it does what they've asked it to do.

Re: Reasoning models don't always say what they think

#107

Post by miven »

xg15 wrote: post #75

> There’s no specific reason why the reported Chain-of-Thought must accurately reflect the true reasoning process; Isn't the whole reason for chain-of-thought that the tokens sort of are the reasoning process? Yes, there is more internal state in the model's hidden layers while it predicts the next token - but that information is gone at the end of that prediction pass. The information that is kept "between one token…

I'm not sure I understand what you're trying to say here, information between tokens is propagated through self-attention, and there's an attention block inside each transformer block within the model, that's a whole lot of internal state that's stored in (mostly) inscrutable key and value vectors with hundreds of dimensions per attention head, around a few dozen heads per attention block, and around a few dozen blocks per model.

Re: Reasoning models don't always say what they think

#108

Post by ertgbnm »

pton_xd wrote: post #21

I was under the impression that CoT works because spitting out more tokens = more context = more compute used to "think." Using CoT as a way for LLMs "show their working" never seemed logical, to me. It's just extra synthetic context.

But the model doesn't have an internal state, it just has the tokens, which means it must encode it's reasoning into the output tokens. So it is a reasonable take to think that CoT was them showing their work.

Re: Reasoning models don't always say what they think

#109

Post by justonenote »

no_wizard wrote: post #100
justonenote wrote: post #78

Earlier quoted context omitted.

> it all rests on (relatively) simple mathematics. We know this is true. We also know that means it has limitations and can't actually reason information. What do you imagine is happening inside biological minds that enables reasoning that is something different to, a lot of, "simple mathematics"? You state that because it is built up of simple mathematics it cannot be reasoning, but this does not follow at all, unle…

Because whats inside our minds is more than mathematics, or we would be able to explain human behavior with the purity of mathematics, and so far, we can't. We can prove the behavior of LLMs with mathematics, because its foundations are constructed. That also means it has the same limits of anything else we use applied mathematics for. Is the broad market analysis that HFT firms use software for to make automated tra…

I mean some people have a definition of intelligence that includes a light switch, it has an internal state, it reacts to external stimuli to affect the world around it, so a light switch is more intelligent than a rock.

Leaving aside where you draw the line of what classifies as intelligence or not , you seem to be invoking some kind of non-materialist view of the human mind, that there is some other 'essence' that is not based on fundamental physics and that is what gives rise to intelligence.

If you subscribe to a materialist world view, that the mind is essentially a biological machine then it has to follow that you can replicate it in software and math. To state otherwise is, as I said, invoking a non-materialistic view that there is something non-physical that gives rise to intelligence.

Re: Reasoning models don't always say what they think

#110

Post by sirsinsalot »

dTal wrote: post #52
lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

>The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM Was it ever seriously entertained? I thought the point was not to reveal a chain of thought, but to produce one. A single token's inference must happen in constant time. But an arbitrarily long chain of tokens can encode an arbitrarily complex chain of reasoning. An LLM is e…

I don't see why a humans internal monologue isn't just a buildup of context to improve pattern matching ahead.

The real answer is... We don't know how much it is or isn't. There's little rigor in either direction.

Post reply on HN