Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

arstechnica.com

Post reply on HN
Print view View original

21–30 of 140 posts

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#21

Post by antihipocrat »

PeterStuer wrote: post #17
Frieren wrote: post #6

This assessment fits with my anecdotal evidence. LLMs just cannot reason in any basic way. LLMs have a large knowledge base that can be spit out at a moment notice. But they have zero insight on its contents, even when the information has just been asked a few lines before. Most of the "intelligence" that LLMs show is just the ability to ask in the correct way the correct questions mirrored back to the user. That is…

Please don't tell me you were one of those marking every SO question as duplicate, more often than not missing the entire nuance in the question that made it not a duplicate at all, and the answers to the so called previously asked question utterly unusable? This was one of those infuriating things that drove so many away from SO and jump ship the second there was an alternative.

I'm not sure why duplicates were ever considered an issue. For certain subjects (like JS) things evolved so quickly during the height of SO that even a year old answer was outdated.

That and search engines seemed to promote more recent content.. so an old answer sank under the ocean of blog spam

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#22

Post by refulgentis »

willvarfar wrote: post #19
NitpickLawyer wrote: post #14

Earlier quoted context omitted.

I have nothing against researching this, I think it's important. My main issue is with articles choosing to grab a "conclusion" and imply it extrapolates to larger models, without any support for that. They are going for the catchy title first, fine-print be damned.

I was just at the KDD conference and the general consensus agreed with this paper. There was only one keynoter who just made the assumption that LLMs are associated with reasoning, which was jarring as the previous keynoter had just explained at length why we need a neuro-symbolic approach instead. The thing is, I think the current companies making LLMs are _not_ trying to be correct or right. They are just trying to…

Not sure what all this is about, I somewhat regret taking a breaking from coding with LLMs to have it explained to me its all a mirage and a secret and sloppy plan for getting me an automagic egirl or something. ;)

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#23

Post by hooskerdu »

Terr_ wrote: post #18
mirekrusin wrote: post #5

Hold on their evaluation tasks are based on rotating letters in text? Isn't this known weak area for token based models?

I think that's the point, really: It's a reliable and reproducible weakness, but also one where the model can be trained to elicit impressive-looking "reasoning" about what the problem is and how it "plans" to overcome it. Then when it fails to apply the "reasoning", that's evidence the artificial expertise we humans perceived or inferred is actually some kind of illusion. Kind of like a a Chinese Room scenario: If t…

Reminds me of a number of grad students I knew who could “talk circles” around all sorts of subjects but failed to ever be able to apply anything.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#24

Post by Terr_ »

hooskerdu wrote: post #23
Terr_ wrote: post #18

Earlier quoted context omitted.

I think that's the point, really: It's a reliable and reproducible weakness, but also one where the model can be trained to elicit impressive-looking "reasoning" about what the problem is and how it "plans" to overcome it. Then when it fails to apply the "reasoning", that's evidence the artificial expertise we humans perceived or inferred is actually some kind of illusion. Kind of like a a Chinese Room scenario: If t…

Reminds me of a number of grad students I knew who could “talk circles” around all sorts of subjects but failed to ever be able to apply anything.

Heh, but just because a human can fail at something doesn't mean everything that fails at it is human. :p

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#25

Post by mdp2021 »

willvarfar wrote: post #19
NitpickLawyer wrote: post #14

Earlier quoted context omitted.

I have nothing against researching this, I think it's important. My main issue is with articles choosing to grab a "conclusion" and imply it extrapolates to larger models, without any support for that. They are going for the catchy title first, fine-print be damned.

I was just at the KDD conference and the general consensus agreed with this paper. There was only one keynoter who just made the assumption that LLMs are associated with reasoning, which was jarring as the previous keynoter had just explained at length why we need a neuro-symbolic approach instead. The thing is, I think the current companies making LLMs are _not_ trying to be correct or right. They are just trying to…

> None of this needs the model to grow strong reasoning skills. That's not where the real money is

"And the world is more and more complex, and the administrations are less and less prepared"

(~~ Henry Kissinger)

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#26

Post by Frieren »

PeterStuer wrote: post #17
Frieren wrote: post #6

This assessment fits with my anecdotal evidence. LLMs just cannot reason in any basic way. LLMs have a large knowledge base that can be spit out at a moment notice. But they have zero insight on its contents, even when the information has just been asked a few lines before. Most of the "intelligence" that LLMs show is just the ability to ask in the correct way the correct questions mirrored back to the user. That is…

Please don't tell me you were one of those marking every SO question as duplicate, more often than not missing the entire nuance in the question that made it not a duplicate at all, and the answers to the so called previously asked question utterly unusable? This was one of those infuriating things that drove so many away from SO and jump ship the second there was an alternative.

I was "playing" the gamification part of StackOverflow. I wanted to ask a good question for points. But it was very difficult because any meaningful question had already been asked. It was way easier to find questions to answer.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#27

Post by kazinator »

NitpickLawyer wrote: post #4

> Without specification, we employ a decoder-only language model GPT2 (Radford et al., 2019) with a configuration of 4 layers, 32 hidden dimensions, and 4 attention heads. Yeah, ok. The research is interesting, warranted, but writing an article about it, and leading with the conclusions gathered from toy models and implying this generalises to production LLMs is useless. We've been here before with small models. Trai…

> conclusions gathered from toy models and implying this generalises to production LLMs is useless

You are just trotting out the tired argument that model size magically fixes the issues, rather than just improves the mirage, and so nothing can be known about models with M parameters by studying models with N Given enough parameters, a miraculous threshold is reached whereby LLMs switch from interpolating to extrapolating.

Sure!

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#28

Post by boredhedgehog »

Terr_ wrote: post #18
mirekrusin wrote: post #5

Hold on their evaluation tasks are based on rotating letters in text? Isn't this known weak area for token based models?

I think that's the point, really: It's a reliable and reproducible weakness, but also one where the model can be trained to elicit impressive-looking "reasoning" about what the problem is and how it "plans" to overcome it. Then when it fails to apply the "reasoning", that's evidence the artificial expertise we humans perceived or inferred is actually some kind of illusion. Kind of like a a Chinese Room scenario: If t…

> Then when it fails to apply the "reasoning", that's evidence the artificial expertise we humans perceived or inferred is actually some kind of illusion.

That doesn't follow, if the weakness of the model manifests on a different level we wouldn't call rational in a human.

For example, a human might have dyslexia, a disorder on the perceptive level. A dyslexic can understand and explain his own limitation, but that doesn't help him overcome it.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#29

Post by [deleted] »

willvarfar wrote: post #10
NitpickLawyer wrote: post #4

> Without specification, we employ a decoder-only language model GPT2 (Radford et al., 2019) with a configuration of 4 layers, 32 hidden dimensions, and 4 attention heads. Yeah, ok. The research is interesting, warranted, but writing an article about it, and leading with the conclusions gathered from toy models and implying this generalises to production LLMs is useless. We've been here before with small models. Trai…

Doing analysis on small models or small data is perfectly valid if the results extrapolate to large models. Which is why right now we're looking at new research papers that are still listing the same small datasets and comparing to the same small models that papers five years ago did.

[deleted]

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#30

Post by kazinator »

NitpickLawyer wrote: post #14
willvarfar wrote: post #10

Earlier quoted context omitted.

Doing analysis on small models or small data is perfectly valid if the results extrapolate to large models. Which is why right now we're looking at new research papers that are still listing the same small datasets and comparing to the same small models that papers five years ago did.

I have nothing against researching this, I think it's important. My main issue is with articles choosing to grab a "conclusion" and imply it extrapolates to larger models, without any support for that. They are going for the catchy title first, fine-print be damned.

Because model size is a trivial parameter, and not a new paradigm.

What you're saying is like, you can't extrapolate that long division works on 100 digit numbers because you only worked through it using 7 digit numbers and a few small polynomials.

Post reply on HN