Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

arstechnica.com

Post reply on HN
Print view View original

61–70 of 140 posts

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#61

Post by intended »

syllogism wrote: post #57
skywhopper wrote: post #55

Earlier quoted context omitted.

So, you agree with the point that they’re making and you’re mad about it? It’s important to state that the models aren’t doing real reasoning because they are being marketed and sold as if they are. As for your question: ‘So what does "sophisticated simulators of reasoning-like text" even mean here?’ It means CoT interstitial “reasoning” steps produce text that looks like reasoning, but is just a rough approximation,…

What is "real reasoning"? The mechanism that the models use is well described. They do what they do. What is this article's complaint?

“the mechanism the models us is well described”

Vs

Total AI capex in the past 6 months was greater than US consumer spending

Or

AGI is coming

Or

AI Agents will be able to do most white collar work

——

The paper is addressing parts of the conversation and expectations of AI that are in the HYPE quadrant. There’s money riding on the idea that AI is going to begin to reason reliably. That it will work as a ghost in the machine.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#62

Post by imp0cat »

zerof1l wrote: post #53

> ... that these "reasoning" models can often produce incoherent, logically unsound answers when questions include irrelevant clauses or deviate even slightly from common templates found in their training data. I have encountered this problem numerous times, now. It really makes me believe that the models do not really understand the topic, even the basics but just try to predict the text. One recent example was me a…

https://www.experimental-history.com/p/bag-of-words-have-mer... Here is an explanation.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#63

Post by kazinator »

ricardobeat wrote: post #33
kazinator wrote: post #27

Earlier quoted context omitted.

> conclusions gathered from toy models and implying this generalises to production LLMs is useless You are just trotting out the tired argument that model size magically fixes the issues, rather than just improves the mirage, and so nothing can be known about models with M parameters by studying models with N Given enough parameters, a miraculous threshold is reached whereby LLMs switch from interpolating to extrapol…

That’s what has been seen in practice though. SOTA LLMs have been shown again and again to solve problems unseen in their data set; and despite their shortcomings they have become extremely useful for a wide variety of tasks.

Even a tiny model for, say, classifying hand-written digits, will correctly classify digits that didn't appear in its training data. (Otherwise it wouldn't be very useful.) That classification is interpolative; the hand-written digit is lands in the space of the training data.

Every result is explainable by has having come from training data. That's the null hypothesis.

The alternative hypothesis is that it's not explainable as having come from training data. That's a hard-to-believe, hard-to-prove negative.

You don't get anything out of any computational process that you didn't put in.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#64

Post by bee_rider »

tankenmate wrote: post #11
suddenlybananas wrote: post #9

Earlier quoted context omitted.

>Training on LLM outputs leads to catastrophic collapse. Every outlet led with this. But no-one red the fine-print, they were testing on small toy models, and were using everything that came out to re-train. Of course it's gonna fail. L3 / phi / gpt-oss models showed that you can absolutely train on synthetic datasets and have great results You're conflating two very different things. Training on synthetic data one t…

"Training on synthetic data one time is very different than cyclically training models on their own data.", but every one with even a modicum of understanding of feedback knows that cyclic training on its own output will end in tears; it's bordering on a tautologic inverse.

Is there an actual general principle or theorem or anything that you can link on this? I’m skeptical because these “model collapse” ideas sound vaguely technical and intuitive, but mostly seem to be based on observations about things that happened to happen with current LLMs. It gets bandied about like it is the most obvious thing, but the support mostly seems to be… pseudo-technical vibes.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#65

Post by Yizahi »

syllogism wrote: post #57
skywhopper wrote: post #55

Earlier quoted context omitted.

So, you agree with the point that they’re making and you’re mad about it? It’s important to state that the models aren’t doing real reasoning because they are being marketed and sold as if they are. As for your question: ‘So what does "sophisticated simulators of reasoning-like text" even mean here?’ It means CoT interstitial “reasoning” steps produce text that looks like reasoning, but is just a rough approximation,…

What is "real reasoning"? The mechanism that the models use is well described. They do what they do. What is this article's complaint?

For example - at minimum reasoning should match what actually happened. This is not even a complete set of criteria for reasoning, but at least a minimal baseline. Currently LLM programs are generating BS in the "reasoning" part of the output. For example ask the LLM program to "reason" how it produces a sum of two numbers and you will see that it doesn't match at all with what LLM program did in the background. The "reasoning" it outputs is simply an extract of the reasoning which humans did in the LLM dataset. Even Anthropic officially admits this. If you ask a program how to do maintenance on a gearbox and it replies with very well articulated and correct (important!) guide to harvest wheat, then we can't call it reasoning of any kind, despite that wheat farming guide was correct and logical.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#66

Post by heresie-dabord »

syllogism wrote: post #16

It's interesting that there's still such a market for this sort of take. > In a recent pre-print paper, researchers from the University of Arizona summarize this existing work as "suggest[ing] that LLMs are not principled reasoners but rather sophisticated simulators of reasoning-like text." What does this even mean? Let's veto the word "reasoning" here and reflect. The LLM produces a series of outputs. Each output c…

> It's interesting that there's still such a market for this sort of take.

What do you think the explanation might be for there being "such a market"?

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#67

Post by nerdjon »

intended wrote: post #61
syllogism wrote: post #57

Earlier quoted context omitted.

What is "real reasoning"? The mechanism that the models use is well described. They do what they do. What is this article's complaint?

“the mechanism the models us is well described” Vs Total AI capex in the past 6 months was greater than US consumer spending Or AGI is coming Or AI Agents will be able to do most white collar work —— The paper is addressing parts of the conversation and expectations of AI that are in the HYPE quadrant. There’s money riding on the idea that AI is going to begin to reason reliably. That it will work as a ghost in the m…

This is why research like this is important and needs to keep being published.

What we have seen the last few years is a conscious marketing effort to rebrand everything ML as AI and to use terms like "Reasoning", "Extended Thinking" and others that for many non technical people give the impression that it is doing far more than it is actually doing.

Many of us here can see his research and be like... well yeah we already knew this. But there is a very well funded effort to oversell what these systems can actually do and that is reaching the people that ultimately make the decisions at companies.

So the question is no longer will AI Agents be able to do most white collar work. They can probably fake it well enough to accomplish a few tasks and management will see that. But will the output actually be valuable long term vs short term gains.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#69

Post by pxc »

NitpickLawyer wrote: post #4

> Without specification, we employ a decoder-only language model GPT2 (Radford et al., 2019) with a configuration of 4 layers, 32 hidden dimensions, and 4 attention heads. Yeah, ok. The research is interesting, warranted, but writing an article about it, and leading with the conclusions gathered from toy models and implying this generalises to production LLMs is useless. We've been here before with small models. Trai…

Well now they could use GPT-OSS, but it wasn't out when they began the study.

I've recently been taking a look at another paper, from 2023, and subsequent research. It has a morally similar finding, though not focused on "reasoning traces", but it's based on GPT-4:

https://proceedings.neurips.cc/paper_files/paper/2023/hash/d...

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#70

Post by [deleted] »

exe34 wrote: post #49
kazinator wrote: post #30

Earlier quoted context omitted.

Because model size is a trivial parameter, and not a new paradigm. What you're saying is like, you can't extrapolate that long division works on 100 digit numbers because you only worked through it using 7 digit numbers and a few small polynomials.

This is 100% how it doesn't work with LLMs.

[deleted]
Post reply on HN