Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

arstechnica.com

Post reply on HN
Print view View original

121–130 of 140 posts

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#121

Post by m3047 »

moi2388 wrote: post #42

“ the researchers created a carefully controlled LLM environment in an attempt to measure just how well chain-of-thought reasoning works when presented with "out of domain" logical problems that don't match the specific logical patterns found in their training data.” Why? If it’s out of domain we know it’ll fail.

There's a question which was rhetorically asked by Yaser S. Abu-Mostafa: "How do we know if we're learning from data?" and his answer was: "We are learning from data if we can generalize from our training set to our problem set."

To me, it feels a lot like Deming's "what gets measured gets done" (with the quiet part "...oftentimes at the expense of everything else."). Of course, the quiet part is different in this case.

What is this "domain" of which you speak? Because LLMs are supposedly good for flying airplanes, mental health, snakebites, and mushroom poisoning.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#122

Post by lossolo »

zwaps wrote: post #93
kazinator wrote: post #30

Earlier quoted context omitted.

Because model size is a trivial parameter, and not a new paradigm. What you're saying is like, you can't extrapolate that long division works on 100 digit numbers because you only worked through it using 7 digit numbers and a few small polynomials.

Scale changes the performance of LLMs. Sometimes, we go so far as to say there is "emergence" of qualitative differences. But really, this is not necessary (and not proven to actually occur). What is true is that the performance of LLMs at OOD tasks changes with scale. So no, it's not the same as solving a math problem.

> What is true is that the performance of LLMs at OOD tasks changes with scale.

If scaling alone guaranteed strong OOD generalization, we’d expect the largest models to consistently top OOD benchmarks but this isn’t the case. In practice, scaling primarily increases a model’s capacity to represent and exploit statistical relationships present in the training distribution. This reliably boosts in-distribution performance but yields limited gains on tasks that are distributionally distant from the training data, especially if the underlying dataset is unchanged. That’s why trillion parameter models trained on the same corpus may excel at tasks similar to those seen in training, but won’t necessarily show proportional improvements on genuinely novel OOD tasks.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#123

Post by throwaway173738 »

intended wrote: post #61
syllogism wrote: post #57

Earlier quoted context omitted.

What is "real reasoning"? The mechanism that the models use is well described. They do what they do. What is this article's complaint?

“the mechanism the models us is well described” Vs Total AI capex in the past 6 months was greater than US consumer spending Or AGI is coming Or AI Agents will be able to do most white collar work —— The paper is addressing parts of the conversation and expectations of AI that are in the HYPE quadrant. There’s money riding on the idea that AI is going to begin to reason reliably. That it will work as a ghost in the m…

The scary thing about ML isn’t that it’s poised to eat a lot of lower-reasoning tasks, it’s that we’re going to find ourselves in a landscape of “that’s just what the AI said to do” kind of excuses for all kinds of bad behavior, and we’re completely unwilling to explore what biases are encoded in the models we’re producing. It’s like how Facebook abdicates responsibility for how users feel because it’s just the product of an algorithm. And if I were a betting person I’d bet all this stuff is going to be used for making rental determinations and for deciding who gets exceptions to overdraft fees well before it’s used for anything else. It’s an enabling technology for all kinds of inhumanity.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#124

Post by anoncareer0212 »

intended wrote: post #59
refulgentis wrote: post #22

Earlier quoted context omitted.

Not sure what all this is about, I somewhat regret taking a breaking from coding with LLMs to have it explained to me its all a mirage and a secret and sloppy plan for getting me an automagic egirl or something. ;)

The point being made doesn’t impact people who can find utility from LLM output. It’s only when you need to apply it to domains outside of code, or a domain where it needs to actually reason, that it becomes an issue.

What does actually reason mean? It's doing this complex anesthesiologist x crna x resident surgery scheduling thingy for ~60 surgeries a day for this one client. Looked a lot like LSAT logic games stuff scaled up to me, took me almost 20-30m to hand check. Is that reasoning?

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#125

Post by Terr_ »

boredhedgehog wrote: post #28
Terr_ wrote: post #18

Earlier quoted context omitted.

I think that's the point, really: It's a reliable and reproducible weakness, but also one where the model can be trained to elicit impressive-looking "reasoning" about what the problem is and how it "plans" to overcome it. Then when it fails to apply the "reasoning", that's evidence the artificial expertise we humans perceived or inferred is actually some kind of illusion. Kind of like a a Chinese Room scenario: If t…

> Then when it fails to apply the "reasoning", that's evidence the artificial expertise we humans perceived or inferred is actually some kind of illusion. That doesn't follow, if the weakness of the model manifests on a different level we wouldn't call rational in a human. For example, a human might have dyslexia, a disorder on the perceptive level. A dyslexic can understand and explain his own limitation, but that d…

I think you're conflating two separate issues: One is the original known impairment that we don't actually care much about, and the other is bullshitting about how the first problem is under-control.

Suppose a real person outlines a viable plan to work-around their dyslexia, and we watch them not do any of it during the test, and they turn in wrong results while describing the workaround they (didn't) follow. This keeps happening over and over.

In that case, we'd probably conclude they have another problem that isn't dyslexia, such as "parroting something they read somewhere and don't really understand."

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#126

Post by famouswaffles »

Terr_ wrote: post #24
hooskerdu wrote: post #23

Earlier quoted context omitted.

Reminds me of a number of grad students I knew who could “talk circles” around all sorts of subjects but failed to ever be able to apply anything.

Heh, but just because a human can fail at something doesn't mean everything that fails at it is human. :p

Right, but if you're saying that something is 'incapable of reasoning' because of a failure mode also found in humans, then either humans are 'incapable of reasoning' or you concede that failure mode isn't a justification for that gross assertion. You can't have it both ways.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#127

Post by willvarfar »

ffsm8 wrote: post #83
willvarfar wrote: post #19

Earlier quoted context omitted.

I was just at the KDD conference and the general consensus agreed with this paper. There was only one keynoter who just made the assumption that LLMs are associated with reasoning, which was jarring as the previous keynoter had just explained at length why we need a neuro-symbolic approach instead. The thing is, I think the current companies making LLMs are _not_ trying to be correct or right. They are just trying to…

> None of this needs the model to grow strong reasoning skills. That's not where the real money is. I never thought about it like that, but it sounds plausible. However, I feel like getting to this stage is even harder to get right compared to reasoning? Aside from the permanence and memory They're currently essentially stateless, while that's surely enough for short term attachment, I'm not seeing this becoming a bi…

(the top post on HN right now is announcing Claude lets you buy a 1M token context. Extrapolate a few years.

Generally, there is a push towards 'context engineering' and there is a lot of bleeding edge research in snapshotting large contexts in ways to get the next back-forth turn in the conversation to be fast etc. So optimisations are already being made.)

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#128

Post by IAmGraydon »

Frieren wrote: post #6

This assessment fits with my anecdotal evidence. LLMs just cannot reason in any basic way. LLMs have a large knowledge base that can be spit out at a moment notice. But they have zero insight on its contents, even when the information has just been asked a few lines before. Most of the "intelligence" that LLMs show is just the ability to ask in the correct way the correct questions mirrored back to the user. That is…

>This assessment fits with my anecdotal evidence. LLMs just cannot reason in any basic way.

Agreed completely, and the sentiment seems to be spreading at an ever-increasing rate. I wonder how long it will be before the bubble collapses. I was thinking maybe as long as a few years, but it might be far sooner at this rate. All it will take is one of the large AI companies coming out and publicly stating that they're no longer making meaningful gains or some other way that shows the public what's really going on behind the curtain.

I'm certain the AI hype bubble will be studied for generations as the greatest mass delusion in history (so far).

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#129

Post by kazinator »

zwaps wrote: post #95
kazinator wrote: post #63

Earlier quoted context omitted.

Even a tiny model for, say, classifying hand-written digits, will correctly classify digits that didn't appear in its training data. (Otherwise it wouldn't be very useful.) That classification is interpolative; the hand-written digit is lands in the space of the training data. Every result is explainable by has having come from training data. That's the null hypothesis. The alternative hypothesis is that it's not exp…

You actually do not classify digits that didn't appear, you classify different pictures of digits that DID appear. Similarly, LLMs do not invent a new way of reasoning about problems or language. They do, however, apply these to unseen problems. LLMs are one level of abstraction up, but it's a very interesting level of abstraction.

No, that is false; a neural net trained on a decent set of handwritten digits will recognize a newly handwritten digit.

I'm saying that this is a strawman version of "not in the training data". The newly handwritten digit is squarely the same sort of stuff that is in the training data: an interpolation.

We are not surprised when we fit a curve to a bunch of points and then find points on the curve that are not exactly any of those points, but are located among the points.

Go too far outside of the cluster of points though and the curve is a hallucination.

This is the intuition behind interpolate vs extrapolate.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#130

Post by wnissen »

syllogism wrote: post #16

It's interesting that there's still such a market for this sort of take. > In a recent pre-print paper, researchers from the University of Arizona summarize this existing work as "suggest[ing] that LLMs are not principled reasoners but rather sophisticated simulators of reasoning-like text." What does this even mean? Let's veto the word "reasoning" here and reflect. The LLM produces a series of outputs. Each output c…

It's not clear what LLMs are good at, and there's great interest in finding out. This is made harder by the frenetic pace of development (GPT 2 came out in 2019). Not surprising at all that there's research into how LLMs fail and why.

Even for someone who kinda understands how the models are trained, it's surprising to me that they struggle when the symbols change. One thing computers are traditionally very good at is symbolic logic. Graph bijection. Stuff like that. So it's worrisome when they fail at it. Even in this research model which is much, much smaller than current or even older models.

Post reply on HN