Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reasoning models don't always say what they think

anthropic.com

Post reply on HN
Print view View original

61–70 of 279 posts

Re: Reasoning models don't always say what they think

#61

Post by Marazan »

tsimionescu wrote: post #47
no_wizard wrote: post #17

Earlier quoted context omitted.

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

One of the earliest things that defined what AI meant were algorithms like A*, and then rules engines like CLIPS. I would say LLMs are much closer to anything that we'd actually call intelligence, despite their limitations, than some of the things that defined* the term for decades. * fixed a typo, used to be "defend"

We had Markov Chains already. Fancy Markov Chains don't seem like a trillion dollar business or actual intelligence.

Re: Reasoning models don't always say what they think

#62

Post by no_wizard »

bigmadshoe wrote: post #45
no_wizard wrote: post #17

Earlier quoted context omitted.

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

We don't have a complete enough theory of neuroscience to conclude that much of human "reasoning" is not "algorithmic pattern matching mixed with statistical likelihoods of success". Regardless of how it models intelligence, why is it not AI? Do you mean it is not AGI? A system that can take a piece of text as input and output a reasonable response is obviously exhibiting some form of intelligence, regardless of the…

It’s easy to attribute intelligence these systems. They have a flexibility and unpredictability that hasn't typically been associated with computers, but it all rests on (relatively) simple mathematics. We know this is true. We also know that means it has limitations and can't actually reason information. The corpus of work is huge - and that allows the results to be pretty striking - but once you do hit a corner with any of this tech, it can't simply reason about the unknown. If its not in the training data - or the training data is outdated - it will not be able to course correct at all. Thus, it lacks reasoning capability, which is a fundamental attribute of any form of intelligence.

Re: Reasoning models don't always say what they think

#64

Post by tsimionescu »

no_wizard wrote: post #50
tsimionescu wrote: post #47

Earlier quoted context omitted.

One of the earliest things that defined what AI meant were algorithms like A*, and then rules engines like CLIPS. I would say LLMs are much closer to anything that we'd actually call intelligence, despite their limitations, than some of the things that defined* the term for decades. * fixed a typo, used to be "defend"

>than some of the things that defend the term for decades There have been many attempts to pervert the term AI, which is a disservice to the technologies and the term itself. Its the simple fact that the business people are relying on what AI invokes in the public mindshare to boost their status and visibility. Thats what bothers me about its misuse so much

Again, if you look at the early papers on AI, you'll see things that are even farther from human intelligence than the LLMs of today. There is no "perversion" of the term, it has always been a vague hypey concept. And it was introduced in this way by academia, not business.

Re: Reasoning models don't always say what they think

#65

Post by nmarinov »

no_wizard wrote: post #48
johnecheck wrote: post #37

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

That's not at all on par with what I'm saying. There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior. We shouldn't seek to muddy this. EDIT: Generally its accepted that a core trait of intelligence is an agent’s ability to achieve goals in a wide range of environments. This means you must be able to generalize, which in turn allows intelligent beings to react to…

I think the confusion is because you're referring to a common understanding of what AI is but I think the definition of AI is different for different people.

Can you give your definition of AI? Also what is the "generally accepted baseline definition for what crosses the threshold of intelligent behavior"?

Re: Reasoning models don't always say what they think

#66

Post by highfrequency »

no_wizard wrote: post #31
dragonwriter wrote: post #15

Earlier quoted context omitted.

> When we get to the point where a LLM can say "oh, I made that mistake because I saw this in my training data, which caused these specific weights to be suboptimal, let me update it", that'll be AGI. While I believe we are far from AGI, I don't think the standard for AGI is an AI doing things a human absolutely cannot do.

We're far from AI. There is no intelligence. The fact the industry decided to move the goal post and re-brand AI for marketing purposes doesn't mean they had a right to hijack a term that has decades of understood meaning. They're using it to bolster the hype around the work, not because there has been a genuine breakthrough in machine intelligence, because there hasn't been one. Now this technology is incredibly use…

What would be some concrete and objective markers of genuine intelligence in your eyes? Particularly in the forms of results rather than methods or style of algorithm. Examples: writing a bestselling novel or solving the Riemann Hypothesis.

Re: Reasoning models don't always say what they think

#67

Post by voidspark »

no_wizard wrote: post #48
johnecheck wrote: post #37

Earlier quoted context omitted.

While I agree that LLMs are hardly sapient, it's very hard to make this argument without being able to pinpoint what a model of intelligence actually is. "Human brains lack any model of intelligence. It's just neurons firing in complicated patterns in response to inputs based on what statistically leads to reproductive success"

That's not at all on par with what I'm saying. There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior. We shouldn't seek to muddy this. EDIT: Generally its accepted that a core trait of intelligence is an agent’s ability to achieve goals in a wide range of environments. This means you must be able to generalize, which in turn allows intelligent beings to react to…

You are doubling down on a muddled vague non-technical intuition about these terms.

Please tell us what that "baseline definition" is.

Re: Reasoning models don't always say what they think

#68

Post by tsimionescu »

Marazan wrote: post #61
tsimionescu wrote: post #47

Earlier quoted context omitted.

One of the earliest things that defined what AI meant were algorithms like A*, and then rules engines like CLIPS. I would say LLMs are much closer to anything that we'd actually call intelligence, despite their limitations, than some of the things that defined* the term for decades. * fixed a typo, used to be "defend"

We had Markov Chains already. Fancy Markov Chains don't seem like a trillion dollar business or actual intelligence.

Completely agree. But if Markov chains are AI (and they always were categorized as such), then fancy Markov chains are still AI.

Re: Reasoning models don't always say what they think

#69

Post by bluefirebrand »

kurthr wrote: post #23
lsy wrote: post #9

The fact that it was ever seriously entertained that a "chain of thought" was giving some kind of insight into the internal processes of an LLM bespeaks the lack of rigor in this field. The words that are coming out of the model are generated to optimize for RLHF and closeness to the training data, that's it! They aren't references to internal concepts, the model is not aware that it's doing anything so how could it…

Yes, but to be fair we're much closer to rationalizing creatures than rational ones. We make up good stories to justify our decisions, but it seems unlikely they are at all accurate.

I would argue that in order to rationalize, you must first be rational

Rationalization is an exercise of (abuse of?) the underlying rational skill

Re: Reasoning models don't always say what they think

#70

Post by no_wizard »

voidspark wrote: post #40
no_wizard wrote: post #17

Earlier quoted context omitted.

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

You are confusing sentience or consciousness with intelligence.

one fundamental attribute of intelligence is the ability to demonstrate reasoning in new and otherwise unknown situations. There is no system that I am currently aware of that works on data it is not trained on.

Another is the fundamental inability to self update on outdated information. It is incapable of doing that, which means it lacks another marker, which is being able to respond to changes of context effectively. Ants can do this. LLMs can't.

Post reply on HN