Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
1–10 of 140 posts
Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
#2The article already seems outdated on the first day. The key points about SFT are irrelevant in the era of RL.
Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
#3The article already seems outdated on the first day. The key points about SFT are irrelevant in the era of RL.
remind me in 2 days
Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
#4> Without specification, we employ a decoder-only language model GPT2 (Radford et al., 2019) with a configuration of 4 layers, 32 hidden dimensions, and 4 attention
heads.
Yeah, ok. The research is interesting, warranted, but writing an article about it, and leading with the conclusions gathered from toy models and implying this generalises to production LLMs is useless.
We've been here before with small models. Training on LLM outputs leads to catastrophic collapse. Every outlet led with this. But no-one red the fine-print, they were testing on small toy models, and were using everything that came out to re-train. Of course it's gonna fail. L3 / phi / gpt-oss models showed that you can absolutely train on synthetic datasets and have great results.
Research in this area is good, and needed. Mainly to understand limitations, discover if there are any scale levels where "emergent" stuff appears and so on. But writing articles based on incipient research, based on tiny models is not worth the effort.
Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
#5Hold on their evaluation tasks are based on rotating letters in text? Isn't this known weak area for token based models?
Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
#6This assessment fits with my anecdotal evidence. LLMs just cannot reason in any basic way.
LLMs have a large knowledge base that can be spit out at a moment notice. But they have zero insight on its contents, even when the information has just been asked a few lines before.
Most of the "intelligence" that LLMs show is just the ability to ask in the correct way the correct questions mirrored back to the user. That is why there is so many advice on how to do "proper prompting".
That and the fact that most questions have already been asked before as anyone that spend some time in StackOverflow back in the day realized. And memory and not reasoning is what is needed to answer them.
Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
#7'Chain-of-thought AI "degrades significantly" when asked to generalize beyond training.' - yeah thanks Captain Obvious.
Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
#8If only we could train people like that to see their reasoning output...
Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
#9> Without specification, we employ a decoder-only language model GPT2 (Radford et al., 2019) with a configuration of 4 layers, 32 hidden dimensions, and 4 attention heads. Yeah, ok. The research is interesting, warranted, but writing an article about it, and leading with the conclusions gathered from toy models and implying this generalises to production LLMs is useless. We've been here before with small models. Trai…
>Training on LLM outputs leads to catastrophic collapse. Every outlet led with this. But no-one red the fine-print, they were testing on small toy models, and were using everything that came out to re-train. Of course it's gonna fail. L3 / phi / gpt-oss models showed that you can absolutely train on synthetic datasets and have great results
You're conflating two very different things. Training on synthetic data one time is very different than cyclically training models on their own data. It has nothing to do with model size.
Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
#10> Without specification, we employ a decoder-only language model GPT2 (Radford et al., 2019) with a configuration of 4 layers, 32 hidden dimensions, and 4 attention heads. Yeah, ok. The research is interesting, warranted, but writing an article about it, and leading with the conclusions gathered from toy models and implying this generalises to production LLMs is useless. We've been here before with small models. Trai…
Doing analysis on small models or small data is perfectly valid if the results extrapolate to large models. Which is why right now we're looking at new research papers that are still listing the same small datasets and comparing to the same small models that papers five years ago did.