Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

211–220 of 279 posts

Re: Reasoning models don't always say what they think

#211

Post by fc417fc802 »

travisjungroth wrote: post #202
jdhwosnhw wrote: post #195

Earlier quoted context omitted.

Peoples’ memories are so short. Ten years ago the “well accepted definition of intelligence” was whether something could pass the Turing test. Now that goalpost has been completely blown out of the water and people are scrabbling to come up with a new one that precludes LLMs. A useful definition of intelligence needs to be measurable, based on inputs/outputs, not internal state. Otherwise you run the risk of dictatin…

I’ve realized while reading these comments my opinions on LLMs being intelligent has significantly increased. Rather than argue any specific test, I believe no one can come up with a text-based intelligence test that 90% of literate adults can pass but the top LLMs fail. This would mean there’s no definition of intelligence you could tie to a test where humans would be intelligent but LLMs wouldn’t. A maybe more pala…

Why presuppose that it's possible to test intelligence via text? Most humans have been illiterate for most of human history.

I don't mean to claim that it isn't possible, just that I'm not clear why we should assume that it is or that there would be an obvious way of going about it.

Re: Reasoning models don't always say what they think

#212

Post by jfengel »

whilenot-dev wrote: post #165
johnecheck wrote: post #37

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

What's wrong with just calling them smart algorithmic models? Being smart allows somewhat to be wrong, as long as that leads to a satisfying solution. Being intelligent on the other hand requires foundational correctness in concepts that aren't even defined yet. EDIT: I also somewhat like the term imperative knowledge (models) [0] [0]: https://en.wikipedia.org/wiki/Procedural_knowledge

The problem with "smart" is that they fail at things that dumb people succeed at. They have ludicrous levels of knowledge and a jaw dropping ability to connect pieces while missing what's right in front of them.

The gap makes me uncomfortable with the implications of the word "smart". It is orthogonal to that.

Re: Reasoning models don't always say what they think

#213

Post by mrob »

travisjungroth wrote: post #204
daveguy wrote: post #192

Earlier quoted context omitted.

The only reason LLMs are stochastic instead of deterministic is a random number generator. There is nothing inherently non-deterministic about LLM algorithms unless you turn up the "temperature" of selecting the next word. The fact that determinism can be changed by turning a knob is clear evidence that they are closer to a database or search engine than a human.

You can turn the determinism knob on humans. Psychedelics are one method.

I think that's more adjusting the parameters of the built-in denoising and feature detection circuits of the inherently noisy analog computer that is the brain.

Re: Reasoning models don't always say what they think

#214

Post by fc417fc802 »

hatefulmoron wrote: post #180
madethisnow wrote: post #156

Earlier quoted context omitted.

datasets and search engines are deterministic. humans, and llms are not.

The LLM's output is chaotic relative to the input, but it's deterministic right? Same settings, same model, same input, .. same output? Where does the chain get broken here?

Now compare a human to an LSTM with persistent internal state that you can't reset.

Re: Reasoning models don't always say what they think

#215

Post by throwanem »

fc417fc802 wrote: post #209
throwanem wrote: post #196

Earlier quoted context omitted.

Mistaking model for meaning is the sort of mistake I very rarely see a human make, at least in the sense as here of literally referring to map ("text"), in what ostensibly strives to be a discussion of the presence or absence of underlying territory, a concept the model gives no sign of attempting to invoke or manipulate. It's also a behavior I would expect from something capable of producing valid utterances but not…

"Concerned on your behalf" seems a bit of an overstatement. Getting caught up on textual representation and failing to notice that the issue is fundamental and generalizes is indeed an error but it's not at all uncharacteristic of even fairly intelligent humans.

All else equal, I wouldn't find it cause for concern. In a discussion where being able to keep the distinction clear in mind at all times absolutely is table stakes, though? I could be fairly blamed for a sprinkle of hyperbole perhaps, but surely you see how an error that is trivial in many contexts would prove so uncommonly severe a flaw in this one, alongside which I reiterate the unusually obtuse nature of the error in this example.

(For those no longer able to follow complex English grammar: Yeah, I exaggerate, but there is no point trying to participate in this kind of discussion if that's the sort of basic error one has to start from, and the especially weird nature of this example of the mistake also points to LLMs synthesizing the result of consciousness rather than experiencing it.)

Re: Reasoning models don't always say what they think

#216

Post by fc417fc802 »

voidspark wrote: post #207
daveguy wrote: post #198

Earlier quoted context omitted.

No. His argument is definitely closer to LLMs can't generalize. I think you would benefit from re-reading the paper. The point is that a puzzle consisting of simple reasoning about simple priors should be a fairly low bar for "intelligence" (necessary but not sufficient). LLMs performs abysmally because they have a very specific purpose trained goal that is different from solving the ARC puzzles. Humans solve these e…

He is setting a bar, saying that that is the "true" generalization. Deep neural networks are definitely performing generalization at a certain level that beats humans at translation or Go, just not at his ARC bar. He may not think it's good enough, but it's still generalization whether he likes it or not.

I'm not convinced either of your examples is generalization. Consider Go. I don't consider a procedural chess engine to be "generalized" in any sense yet a decent one can easily beat any human. Why then should Go be different?

Re: Reasoning models don't always say what they think

#217

Post by wahnfrieden »

reaperman wrote: post #90
wahnfrieden wrote: post #33

Earlier quoted context omitted.

No, the CoT is not simply extra context the models are specifically trained to use CoT and that includes treating it as unspoken thought

Huge thank you for correcting me. Do you have any good resources I could look at to learn how the previous CoT is included in the input tokens and treated differently?

I've only read the marketing materials of closed models. So they could be lying, too. But I don't think CoT is something you can do with pre-CoT models via prompting and context manipulation. You can do something that looks a little like CoT, but the model won't have been trained specifically on how to make good use of it and will treat it like Q&A context.

Re: Reasoning models don't always say what they think

#218

Post by EncomLab »

The use of highly anthropomorphic language is always problematic- Does a photo resistor controlled nightlight have a chain of thought? Does it reason about its threshold value? Does it have an internal model of what is light, what is dark, and the role it plays in demarcation between the two?

Are the transistors executing the code within the confines even capable of intentionality? If so - where is it derived from?

Re: Reasoning models don't always say what they think

#219

Post by voidspark »

fc417fc802 wrote: post #216
voidspark wrote: post #207

Earlier quoted context omitted.

He is setting a bar, saying that that is the "true" generalization. Deep neural networks are definitely performing generalization at a certain level that beats humans at translation or Go, just not at his ARC bar. He may not think it's good enough, but it's still generalization whether he likes it or not.

I'm not convinced either of your examples is generalization. Consider Go. I don't consider a procedural chess engine to be "generalized" in any sense yet a decent one can easily beat any human. Why then should Go be different?

A procedural chess engine does not perform generalization, in ML terms. That is an explicitly programmed algorithm.

Generalization has a specific meaning in the context of machine learning.

The AlphaGo Zero model learned advanced strategies of the game, starting with only the basic rules of the game, without being programmed explicitly. That is generalization.

Re: Reasoning models don't always say what they think

#220

Post by arctek »

no_wizard wrote: post #17
lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

This is also why I think the current iterations wont converge on any actual type of intelligence.

It doesn't operate on the same level as (human) intelligence it's a very path dependent process. Every step you add down this path increases entropy as well and while further improvements and bigger context windows help - eventually you reach a dead end where it degrades.

You'd almost need every step of the process to mutate the model to update global state from that point.

From what I've seen the major providers kind of use tricks to accomplish this, but it's not the same thing.

Post reply on HN