Reasoning models don't always say what they think • Hacker News Board

Re: Reasoning models don't always say what they think

#81

Post by danielbln » Thu, Apr 03, 2025, 7:03 PM UTC

bigmadshoe wrote: post #45

no_wizard wrote: post #17

Earlier quoted context omitted.

>internal concepts, the model is not aware that it's doing anything so how could it "explain itself" This in a nutshell is why I hate that all this stuff is being labeled as AI. Its advanced machine learning (another term that also feels inaccurate but I concede is at least closer to whats happening conceptually) Really, LLMs and the like still lack any model of intelligence. Its, in the most basic of terms, algorith…

We don't have a complete enough theory of neuroscience to conclude that much of human "reasoning" is not "algorithmic pattern matching mixed with statistical likelihoods of success". Regardless of how it models intelligence, why is it not AI? Do you mean it is not AGI? A system that can take a piece of text as input and output a reasonable response is obviously exhibiting some form of intelligence, regardless of the…

I always wonder where people get their confidence from. We know so little about our own cognition, what makes us tick, how consciousness emerges, how about thought processes actually fundamentally work. We don't even know why we dream. Yet people proclaim loudly that X clearly isn't intelligent. Ok, but based on what?

Re: Reasoning models don't always say what they think

#82

Post by dingnuts » Thu, Apr 03, 2025, 7:05 PM UTC

byearthithatius wrote: post #57

no_wizard wrote: post #48

Earlier quoted context omitted.

That's not at all on par with what I'm saying. There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior. We shouldn't seek to muddy this. EDIT: Generally its accepted that a core trait of intelligence is an agent’s ability to achieve goals in a wide range of environments. This means you must be able to generalize, which in turn allows intelligent beings to react to…

"There exists a generally accepted baseline definition for what crosses the threshold of intelligent behavior" not really. The whole point they are trying to make is that the capability of these models IS ALREADY muddying the definition of intelligence. We can't really test it because the distribution its learned is so vast. Hence why he have things like ARC now. Even if its just gradient descent based distribution l…

How does an LLM muddy the definition of intelligence any more than a database or search engine does? They are lossy databases with a natural language interface, nothing more.

Re: Reasoning models don't always say what they think

#83

Post by lpzimm » Thu, Apr 03, 2025, 7:06 PM UTC

Not exactly the same as this study, but I'll ask questions to LLMs with and without subtle hints to see if it changes the answer and it almost always does. For example, paraphrased:

No hint: "I have an otherwise unused variable that I want to use to record things for the debugger, but I find it's often optimized out. How do I prevent this from happening?"

Answer: 1. Mark it as volatile (...)

Hint: "I have an otherwise unused variable that I want to use to record things for the debugger, but I find it's often optimized out. Can I solve this with the volatile keyword or is that a misconception?"

Answer: Using volatile is a common suggestion to prevent optimizations, but it does not guarantee that an unused variable will not be optimized out. Try (...)

This is Claude 3.7 Sonnet.

Re: Reasoning models don't always say what they think

#84

Post by svachalek » Thu, Apr 03, 2025, 7:07 PM UTC

Marazan wrote: post #61

tsimionescu wrote: post #47

Earlier quoted context omitted.

One of the earliest things that defined what AI meant were algorithms like A*, and then rules engines like CLIPS. I would say LLMs are much closer to anything that we'd actually call intelligence, despite their limitations, than some of the things that defined* the term for decades. * fixed a typo, used to be "defend"

We had Markov Chains already. Fancy Markov Chains don't seem like a trillion dollar business or actual intelligence.

An LLM is no more a fancy Markov Chain than you are. The math is well documented, go have a read.

Re: Reasoning models don't always say what they think

#85

Post by nopelynopington » Thu, Apr 03, 2025, 7:08 PM UTC

Of course they don't.

LLMs are a brainless algorithm that guesses the next word. When you ask them what they think they're also guessing the next word. No reason for it to match, except a trick of context

Re: Reasoning models don't always say what they think

#86

Post by [deleted] » Thu, Apr 03, 2025, 7:09 PM UTC

[deleted]

Re: Reasoning models don't always say what they think

#87

Post by AgentME » Thu, Apr 03, 2025, 7:09 PM UTC

ctoth wrote: post #51

I invite anyone who postulates humans are more than just "spicy autocomplete" to examine this thread. The level of actual reasoning/engaging with the article is ... quite something.

Internet commenters don't "reason". They just generate inane arguments over definitions, like a lowly markov bot, without the true spark of life and soul that even certain large language models have.

Re: Reasoning models don't always say what they think

#88

Post by svachalek » Thu, Apr 03, 2025, 7:12 PM UTC

xg15 wrote: post #75

> There’s no specific reason why the reported Chain-of-Thought must accurately reflect the true reasoning process; Isn't the whole reason for chain-of-thought that the tokens sort of are the reasoning process? Yes, there is more internal state in the model's hidden layers while it predicts the next token - but that information is gone at the end of that prediction pass. The information that is kept "between one token…

Exactly. There's no state outside the context. The difference in performance between the non-reasoning model and the reasoning model comes from the extra tokens in the context. The relationship isn't strictly a logical one, just as it isn't for non-reasoning LLMs, but the process is autoregression and happens in plain sight.

Re: Reasoning models don't always say what they think

#89

Post by semiquaver » Thu, Apr 03, 2025, 7:15 PM UTC

vohk wrote: post #30

semiquaver wrote: post #16

Earlier quoted context omitted.

That’s holding LLMs to a significantly higher standard than humans. When I realize there’s a flaw in my reasoning I don’t know that it was caused by specific incorrect neuron connections or activation potentials in my brain, I think of the flaw in domain-specific terms using language or something like it. Outputting CoT content, thereby making it part of the context from which future tokens will be generated, is roug…

I think you're anthropomorphizing there. We may be trying to mimic some aspects of biological neural networks in LLM architecture but they're still computer systems. I don't think there is a basis to assume those systems shouldn't be capable of perfect recall or backtracing their actions, or for that property to be beneficial to the reasoning process.

Of course I’m anthropomorphizing. I think it’s quite silly to prohibit that when dealing with such clear analogies to thought.

Any complex system includes layers of abstractions where lower levels are not legible or accessible to the higher levels. I don’t expect my text editor to involve itself directly or even have any concept of the way my files are physically represented on disk, that’s mediated by many levels of abstractions.

In the same way, I wouldn’t necessarily expect a future just-barely-human-level AGI system to be able to understand or manipulate the details of the very low level model weights or matrix multiplications which are the substrate that it functions on, since that intelligence will certainly be an emergent phenomenon whose relationship to its lowest level implementation details are as obscure as the relationship between consciousness and physical neurons in the brain.

Re: Reasoning models don't always say what they think

#90

Post by reaperman » Thu, Apr 03, 2025, 7:17 PM UTC

wahnfrieden wrote: post #33

reaperman wrote: post #11

Earlier quoted context omitted.

Edit: 'wahnfrieden corrected me. I incorrectly posited that CoT was only included in the context window during the reasoning task and later left out entirely. Edited to remove potential misinformation.

No, the CoT is not simply extra context the models are specifically trained to use CoT and that includes treating it as unspoken thought

Huge thank you for correcting me. Do you have any good resources I could look at to learn how the previous CoT is included in the input tokens and treated differently?