Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

arstechnica.com

Post reply on HN
Print view View original

111–120 of 140 posts

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#111

Post by bumby »

zwaps wrote: post #95
kazinator wrote: post #63

Earlier quoted context omitted.

Even a tiny model for, say, classifying hand-written digits, will correctly classify digits that didn't appear in its training data. (Otherwise it wouldn't be very useful.) That classification is interpolative; the hand-written digit is lands in the space of the training data. Every result is explainable by has having come from training data. That's the null hypothesis. The alternative hypothesis is that it's not exp…

You actually do not classify digits that didn't appear, you classify different pictures of digits that DID appear. Similarly, LLMs do not invent a new way of reasoning about problems or language. They do, however, apply these to unseen problems. LLMs are one level of abstraction up, but it's a very interesting level of abstraction.

>you classify different pictures of digits that DID appear.

Are you implying models that classify hand-written digits don’t generalize and only work on training data?

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#112

Post by Insanity »

calf wrote: post #54
willvarfar wrote: post #19

Earlier quoted context omitted.

I was just at the KDD conference and the general consensus agreed with this paper. There was only one keynoter who just made the assumption that LLMs are associated with reasoning, which was jarring as the previous keynoter had just explained at length why we need a neuro-symbolic approach instead. The thing is, I think the current companies making LLMs are _not_ trying to be correct or right. They are just trying to…

As to general consensus, Hinton gave a recent talk, and he seemed adamant that neural networks (which LLMs are) really are doing reasoning. He gives his reasons for it. Is Hinton considered an outlier or?

Link to the talk?

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#113

Post by lm28469 »

megaloblasto wrote: post #100
lm28469 wrote: post #91

Earlier quoted context omitted.

And how much of productivity loss due to the insane amount of noise being generated ? (filler ridden reports, emails, videos, podcasts, &c.)

I'm talking about 100% net gain in productivity.

You're talking about net gains in "coding tasks" productivity, I'm talking in productivity gain across the board.

My company deals with an insane amount of customers who use chatgpt to pre-debug their problems before coming to our support. Once they contact our support they regurgitate llm generated BS to our support engineers thinking they're going to speed up the process, the only thing they're doing is generating noise that slows everyone down because chatgpt has absolutely no clue about our product and keeps sending them on wild goose chases. Sometimes they even lie pretending "a colleague" steered them in this or that direction while it's 100% obvious the whole thing was hallucinate and even written by an llm.

I can't tell you how frustrating it is to read a 10 min long customer email just to realise it's just an llm hallucinating probable causes for a bug that takes 2 sentences to describe.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#114

Post by sachin_rcz »

afro88 wrote: post #51

I have a real world problem I gave o1 when it came out and it got it quite wrong. It's a scheduling problem with 4 different constraints that vary each day, and success criteria that need to be fulfilled over the whole week. GPT-5 Thinking (Think Longer) and Opus 4.1 Extended Thinking both get it right. Maybe this unique problem is somehow a part of synthetic training data? Or maybe it's not and the paper is wrong? E…

Models today also have access to certain tooling or have been reinforced to use that tooling in complicated situations. i.e. Questions of counting letters in word are being answered by using python code in background.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#115

Post by PeterStuer »

ceejayoz wrote: post #41
MoreQARespect wrote: post #40

Earlier quoted context omitted.

That's a valid goal, but they should have adapted the software to the community instead of trying to adapt the community to the software. SO's biggest asset was its community and while they treated it with some respect in the beginning they took it for granted and trashed it later.

I think this policy was, in large part, intended to respect the user base, who get exhausted answering the same question over and over. I do agree they later trashed that relationship with the Monica incident and AI policies.

Sounds like they optimised for a select 1% class of self appointed gatekeepers rather than the broad user base. Classic mistake of nearly every defunct social site.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#116

Post by megaloblasto »

lm28469 wrote: post #113
megaloblasto wrote: post #100

Earlier quoted context omitted.

I'm talking about 100% net gain in productivity.

You're talking about net gains in "coding tasks" productivity, I'm talking in productivity gain across the board. My company deals with an insane amount of customers who use chatgpt to pre-debug their problems before coming to our support. Once they contact our support they regurgitate llm generated BS to our support engineers thinking they're going to speed up the process, the only thing they're doing is generating…

I agree with that idea. For more business development areas, AI slop can slow things down.

I do think that these kinks will eventually work themselves out and actually increase productivity in these areas. People also need to learn that it is not acceptable to just generate some BS and send it to your boss or colleague. That just transfers the real work of understanding the generated content to someone else.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#118

Post by moi2388 »

podgorniy wrote: post #56
moi2388 wrote: post #42

“ the researchers created a carefully controlled LLM environment in an attempt to measure just how well chain-of-thought reasoning works when presented with "out of domain" logical problems that don't match the specific logical patterns found in their training data.” Why? If it’s out of domain we know it’ll fail.

> Why? If it’s out of domain we know it’ll fail. To see if LLMs adhere to logic or observed "logical" responses are rather reproduction of patterns. I personally enjoy this idea of isolation "logic" from "pattern" and seeing if "logic" will manifest in LLM "thinking" about in "non-patternized" domain. -- Also it's never bad give proves to public that "thinking" (like "intelligence") in AI context isn't the same thing…

Out of domain means that the type of logic hasn’t been in the training set.

“All A are B, All C are D, X is A and B, what is X?” is not outside this domain.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#119

Post by pxc »

Xss3 wrote: post #74
Yizahi wrote: post #65

Earlier quoted context omitted.

For example - at minimum reasoning should match what actually happened. This is not even a complete set of criteria for reasoning, but at least a minimal baseline. Currently LLM programs are generating BS in the "reasoning" part of the output. For example ask the LLM program to "reason" how it produces a sum of two numbers and you will see that it doesn't match at all with what LLM program did in the background. The…

As soon as you introduce multiple constraints on what is and isn't reasoning people get confused and disengage. I like this approach of setting a minimum constraint. But i feel adding more will just make people ignore the point entirely.

The reality is obvious. The only way not to see it when looking at research like this is to not want to see it. The idea that this critique is somehow more confusing than the use of the word "reasoning" itself is farcical.

LLMs are cool and some of the things they can do now are useful, even surprising. But when it comes to AI, business leaders are talking their books and many people are swept up by that breathless talk and their own misleading intuitions, frequently parroted by the media.

The "but human reasoning is also flawed, so I can't possibly understand what you mean!" objection cannot be sustained in good faith short of delusion.

Re: Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens

#120

Post by staticman2 »

zwaps wrote: post #96
boxed wrote: post #88

Earlier quoted context omitted.

> SOTA LLMs have been shown again and again to solve problems unseen in their data set We have no idea what the training data is though, so you can't say that. > and despite their shortcomings they have become extremely useful for a wide variety of tasks. That seems like a separate question.

I have applied O3 pro on unpublished abandoned research of mine that was never published and lives in an intersection that is as entirely novel as it's uninteresting. O3 pro (but not O3) was successfully able to apply reasoning and math to this domain in interesting ways, much like an expert researcher in these areas would. Again, the field and the problem is with 100% certainty OOD of the data. However, the techniqu…

The paper is evaluating how well an LLM can handle novelty, and on the paper's terms you need to calculate or otherwise somehow deduce the degree or type of novelty rather than simply describing your never published research as novel.

I don't even know that this is possible without seeing the training data. Hence the difficulty in describing how good at "reasoning" O3 Pro is.

The most novel problem would presumably be something only a martian could understand, written in an alien language, the least novel problem would be a basic question taught in preschool like what color is the sky.

Your research falls somewhere between those extremes.

Post reply on HN