Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

arstechnica.com

Post reply on HN
Print view View original

71–80 of 140 posts

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#71

Post by kevingadd »

NitpickLawyer wrote: post #4

> Without specification, we employ a decoder-only language model GPT2 (Radford et al., 2019) with a configuration of 4 layers, 32 hidden dimensions, and 4 attention heads. Yeah, ok. The research is interesting, warranted, but writing an article about it, and leading with the conclusions gathered from toy models and implying this generalises to production LLMs is useless. We've been here before with small models. Trai…

Almost every mention I've seen of gpt-oss was a complaint that the training on synthetic datasets produced a model that's mostly good at benchmarks. Are benchmarks the great results you're referring to or are there a lot of satisfied users out there that just don't post here on HN? Genuinely curious.

I can see how performing well on benchmarks at the expense of everything else counts as great results if that's the point of the model.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#72

Post by kevingadd »

boredhedgehog wrote: post #28
Terr_ wrote: post #18

Earlier quoted context omitted.

I think that's the point, really: It's a reliable and reproducible weakness, but also one where the model can be trained to elicit impressive-looking "reasoning" about what the problem is and how it "plans" to overcome it. Then when it fails to apply the "reasoning", that's evidence the artificial expertise we humans perceived or inferred is actually some kind of illusion. Kind of like a a Chinese Room scenario: If t…

> Then when it fails to apply the "reasoning", that's evidence the artificial expertise we humans perceived or inferred is actually some kind of illusion. That doesn't follow, if the weakness of the model manifests on a different level we wouldn't call rational in a human. For example, a human might have dyslexia, a disorder on the perceptive level. A dyslexic can understand and explain his own limitation, but that d…

Typically when a human has a disorder or limitation they adapt to it by developing coping strategies or making use of tools and environmental changes to compensate. Maybe they expect a true reasoning model to be able to do the same thing?

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#73

Post by [deleted] »

logicchains wrote: post #45
willvarfar wrote: post #10

Earlier quoted context omitted.

Doing analysis on small models or small data is perfectly valid if the results extrapolate to large models. Which is why right now we're looking at new research papers that are still listing the same small datasets and comparing to the same small models that papers five years ago did.

The extrapolation doesn't work if the transformer is too shallow (too few layers) relative to sequence length, because of https://arxiv.org/abs/2503.03961 . A bunch of tasks become unfeasible when the layer count is too low, and 4 layers is way too low. I.e. linearly increasing the number of layers in a model can result in a superlinear increase in performance on tasks like reasoning.

[deleted]

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#74

Post by Xss3 »

Yizahi wrote: post #65
syllogism wrote: post #57

Earlier quoted context omitted.

What is "real reasoning"? The mechanism that the models use is well described. They do what they do. What is this article's complaint?

For example - at minimum reasoning should match what actually happened. This is not even a complete set of criteria for reasoning, but at least a minimal baseline. Currently LLM programs are generating BS in the "reasoning" part of the output. For example ask the LLM program to "reason" how it produces a sum of two numbers and you will see that it doesn't match at all with what LLM program did in the background. The…

As soon as you introduce multiple constraints on what is and isn't reasoning people get confused and disengage.

I like this approach of setting a minimum constraint. But i feel adding more will just make people ignore the point entirely.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#75

Post by sdenton4 »

skywhopper wrote: post #55
syllogism wrote: post #16

It's interesting that there's still such a market for this sort of take. > In a recent pre-print paper, researchers from the University of Arizona summarize this existing work as "suggest[ing] that LLMs are not principled reasoners but rather sophisticated simulators of reasoning-like text." What does this even mean? Let's veto the word "reasoning" here and reflect. The LLM produces a series of outputs. Each output c…

So, you agree with the point that they’re making and you’re mad about it? It’s important to state that the models aren’t doing real reasoning because they are being marketed and sold as if they are. As for your question: ‘So what does "sophisticated simulators of reasoning-like text" even mean here?’ It means CoT interstitial “reasoning” steps produce text that looks like reasoning, but is just a rough approximation,…

"the reasoning often doesn’t line up with the conclusion, or the priors, or reality."

My dude, have you ever interacted with human reasoning?

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#77

Post by NoGravitas »

ponow wrote: post #76

> LLMs are [...] sophisticated simulators of reasoning-like text Most humans are unsophisticated simulators of reasoning-like text.

Except you, right? You're one of the special few who can actually reason, not like /those/ people.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#78

Post by dcre »

loosetypes wrote: post #36
ricardobeat wrote: post #33

Earlier quoted context omitted.

That’s what has been seen in practice though. SOTA LLMs have been shown again and again to solve problems unseen in their data set; and despite their shortcomings they have become extremely useful for a wide variety of tasks.

Mind linking any examples (or categories) of problems that are definitively not in pre training data but can still be solved by LLMs? Preferably something factual rather than creative, genuinely curious. Dumb question but anything like this that’s written about on the internet will ultimately end up as training fodder, no?

How about the International Math Olympiad?

https://arstechnica.com/ai/2025/07/google-deepmind-earns-gol...

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#79

Post by ponow »

wzdd wrote: post #48
syllogism wrote: post #16

It's interesting that there's still such a market for this sort of take. > In a recent pre-print paper, researchers from the University of Arizona summarize this existing work as "suggest[ing] that LLMs are not principled reasoners but rather sophisticated simulators of reasoning-like text." What does this even mean? Let's veto the word "reasoning" here and reflect. The LLM produces a series of outputs. Each output c…

> The mechanism that the model uses to transition towards the answer is to generate intermediate text. Yes, which makes sense, because if there's a landscape of states that the model is traversing, and there are probablistically likely pathways between an initial state and the desired output, but there isn't a direct pathway, then training the the model to generate intermediate text in order to move across that lands…

Meaningless? The participation in a usefully predicting path is meaning. A different meaning.

And Gemini has a note at the bottom about mistakes, and many people discuss this. Caveat emptor, as usual.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#80

Post by ponow »

nerdjon wrote: post #67
intended wrote: post #61

Earlier quoted context omitted.

“the mechanism the models us is well described” Vs Total AI capex in the past 6 months was greater than US consumer spending Or AGI is coming Or AI Agents will be able to do most white collar work —— The paper is addressing parts of the conversation and expectations of AI that are in the HYPE quadrant. There’s money riding on the idea that AI is going to begin to reason reliably. That it will work as a ghost in the m…

This is why research like this is important and needs to keep being published. What we have seen the last few years is a conscious marketing effort to rebrand everything ML as AI and to use terms like "Reasoning", "Extended Thinking" and others that for many non technical people give the impression that it is doing far more than it is actually doing. Many of us here can see his research and be like... well yeah we al…

I'm happy enough if I'm better off for having used a tool than having not.
Post reply on HN