Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

arstechnica.com

Post reply on HN
Print view View original

51–60 of 140 posts

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#51

Post by afro88 »

I have a real world problem I gave o1 when it came out and it got it quite wrong. It's a scheduling problem with 4 different constraints that vary each day, and success criteria that need to be fulfilled over the whole week.

GPT-5 Thinking (Think Longer) and Opus 4.1 Extended Thinking both get it right.

Maybe this unique problem is somehow a part of synthetic training data? Or maybe it's not and the paper is wrong? Either way, we have models that are much more capable at solving unique problems today.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#52

Post by barrkel »

kazinator wrote: post #30
NitpickLawyer wrote: post #14

Earlier quoted context omitted.

I have nothing against researching this, I think it's important. My main issue is with articles choosing to grab a "conclusion" and imply it extrapolates to larger models, without any support for that. They are going for the catchy title first, fine-print be damned.

Because model size is a trivial parameter, and not a new paradigm. What you're saying is like, you can't extrapolate that long division works on 100 digit numbers because you only worked through it using 7 digit numbers and a few small polynomials.

Alas, not true. It would be easier to predict progress if so.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#53

Post by zerof1l »

> ... that these "reasoning" models can often produce incoherent, logically unsound answers when questions include irrelevant clauses or deviate even slightly from common templates found in their training data.

I have encountered this problem numerous times, now. It really makes me believe that the models do not really understand the topic, even the basics but just try to predict the text.

One recent example was me asking the model to fix my docker-compose file. In it, there's the `network: host` for the `build` part. The model kept assuming that the container would be running with the host network and kept asking me to remove it as a way to fix my issue, even though it wouldn't do anything for the container that is running. Because container runs on `custom_net` network only. The model was obsessed with it and kept telling me to remove it until I explicitly told that it is not, and cannot be the issue.

``` services:

  app:

    build:

      network: host

    networks:

      custom_net:

    ...
```

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#54

Post by calf »

willvarfar wrote: post #19
NitpickLawyer wrote: post #14

Earlier quoted context omitted.

I have nothing against researching this, I think it's important. My main issue is with articles choosing to grab a "conclusion" and imply it extrapolates to larger models, without any support for that. They are going for the catchy title first, fine-print be damned.

I was just at the KDD conference and the general consensus agreed with this paper. There was only one keynoter who just made the assumption that LLMs are associated with reasoning, which was jarring as the previous keynoter had just explained at length why we need a neuro-symbolic approach instead. The thing is, I think the current companies making LLMs are _not_ trying to be correct or right. They are just trying to…

As to general consensus, Hinton gave a recent talk, and he seemed adamant that neural networks (which LLMs are) really are doing reasoning. He gives his reasons for it. Is Hinton considered an outlier or?

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#55

Post by skywhopper »

syllogism wrote: post #16

It's interesting that there's still such a market for this sort of take. > In a recent pre-print paper, researchers from the University of Arizona summarize this existing work as "suggest[ing] that LLMs are not principled reasoners but rather sophisticated simulators of reasoning-like text." What does this even mean? Let's veto the word "reasoning" here and reflect. The LLM produces a series of outputs. Each output c…

So, you agree with the point that they’re making and you’re mad about it? It’s important to state that the models aren’t doing real reasoning because they are being marketed and sold as if they are.

As for your question: ‘So what does "sophisticated simulators of reasoning-like text" even mean here?’

It means CoT interstitial “reasoning” steps produce text that looks like reasoning, but is just a rough approximation, given that the reasoning often doesn’t line up with the conclusion, or the priors, or reality.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#56

Post by podgorniy »

moi2388 wrote: post #42

“ the researchers created a carefully controlled LLM environment in an attempt to measure just how well chain-of-thought reasoning works when presented with "out of domain" logical problems that don't match the specific logical patterns found in their training data.” Why? If it’s out of domain we know it’ll fail.

> Why? If it’s out of domain we know it’ll fail.

To see if LLMs adhere to logic or observed "logical" responses are rather reproduction of patterns.

I personally enjoy this idea of isolation "logic" from "pattern" and seeing if "logic" will manifest in LLM "thinking" about in "non-patternized" domain.

--

Also it's never bad give proves to public that "thinking" (like "intelligence") in AI context isn't the same thing we think about intuitively.

--

> If it’s out of domain we know it’ll fail.

Below goes question which is out of domain. Yet LLMs handle the replies in what appearing as logical way.

``` Kookers are blight. And shmakers are sin. If peker is blight and sin who is he? ```

It is out of domain and it does not fail (I've put it through thinking gemini 2.5). Now back to article. Is observed logic intristic to LLMs or it's an elaborate form of a pattern? Acoording to article it's a pattern.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#57

Post by syllogism »

skywhopper wrote: post #55
syllogism wrote: post #16

It's interesting that there's still such a market for this sort of take. > In a recent pre-print paper, researchers from the University of Arizona summarize this existing work as "suggest[ing] that LLMs are not principled reasoners but rather sophisticated simulators of reasoning-like text." What does this even mean? Let's veto the word "reasoning" here and reflect. The LLM produces a series of outputs. Each output c…

So, you agree with the point that they’re making and you’re mad about it? It’s important to state that the models aren’t doing real reasoning because they are being marketed and sold as if they are. As for your question: ‘So what does "sophisticated simulators of reasoning-like text" even mean here?’ It means CoT interstitial “reasoning” steps produce text that looks like reasoning, but is just a rough approximation,…

What is "real reasoning"? The mechanism that the models use is well described. They do what they do. What is this article's complaint?

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#58

Post by [deleted] »

exe34 wrote: post #49
kazinator wrote: post #30

Earlier quoted context omitted.

Because model size is a trivial parameter, and not a new paradigm. What you're saying is like, you can't extrapolate that long division works on 100 digit numbers because you only worked through it using 7 digit numbers and a few small polynomials.

This is 100% how it doesn't work with LLMs.

[deleted]

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#59

Post by intended »

refulgentis wrote: post #22
willvarfar wrote: post #19

Earlier quoted context omitted.

I was just at the KDD conference and the general consensus agreed with this paper. There was only one keynoter who just made the assumption that LLMs are associated with reasoning, which was jarring as the previous keynoter had just explained at length why we need a neuro-symbolic approach instead. The thing is, I think the current companies making LLMs are _not_ trying to be correct or right. They are just trying to…

Not sure what all this is about, I somewhat regret taking a breaking from coding with LLMs to have it explained to me its all a mirage and a secret and sloppy plan for getting me an automagic egirl or something. ;)

The point being made doesn’t impact people who can find utility from LLM output.

It’s only when you need to apply it to domains outside of code, or a domain where it needs to actually reason, that it becomes an issue.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#60

Post by drawfloat »

bubblyworld wrote: post #34
syllogism wrote: post #16

It's interesting that there's still such a market for this sort of take. > In a recent pre-print paper, researchers from the University of Arizona summarize this existing work as "suggest[ing] that LLMs are not principled reasoners but rather sophisticated simulators of reasoning-like text." What does this even mean? Let's veto the word "reasoning" here and reflect. The LLM produces a series of outputs. Each output c…

Not sure why everyone is downvoting you as I think you raise a good point - these anthropomorphic words like "reasoning" are useful as shorthands for describing patterns of behaviour, and are generally not meant to be direct comparisons to human cognition. But it goes both ways. You can still criticise the model on the grounds that what we call "reasoning" in the context of LLMs doesn't match the patterns we associat…

""Sam Altman says the perfect AI is “a very tiny model with superhuman reasoning".""

It is being marketed as directly related to human reasoning.

Post reply on HN