Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is chain-of-thought AI reasoning a mirage?

seangoedecke.com

Post reply on HN
Print view View original

181–190 of 191 posts

Re: Is chain-of-thought AI reasoning a mirage?

#181

Post by xg15 »

_diyar wrote: post #114
safety1st wrote: post #87

Earlier quoted context omitted.

I'm pretty much a layperson in this field, but I don't understand why we're trying to teach a stochastic text transformer to reason. Why would anyone expect that approach to work? I would have thought the more obvious approach would be to couple it to some kind of symbolic logic engine. It might transform plain language statements into fragments conforming to a syntax which that engine could then parse deterministica…

We expect this approach to work because it's currently the best working approach. Nothing else comes close. Using symbolic language is a good idea in theory, but in practice it doesn't scale as well as auto-regression + RL. The IMO results of DeepMind illustrate this well: In 2024, they solved it using AlphaProof and AlphaGeometry, using the Lean language as a formal symbolic logic[1]. In 2025 they performed better a…

The example is a bit of an unfair comparison though. It takes questions already posed in natural language and, as far as I can tell, expects results in natural language, too.

This means that whatever system is evaluated in this challenge necessarily has to deal with natural language. And indeed, a big part of the AlphaProof system was a neural network to convert from natural language to Lean.

None of this has anything to do with reasoning ability.

I think it would be interesting to present an inverse challenge, where the problems are already posed in a formal language. Would a network that first converts them into natural language, then does chain-of-thought on that, then translates the result back into formal language still be better than a simple symbolic reasoner that could operate on the formal language directly?

Re: Is chain-of-thought AI reasoning a mirage?

#182

Post by horizion2025 »

safety1st wrote: post #87
LudwigNagasena wrote: post #75

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

I'm pretty much a layperson in this field, but I don't understand why we're trying to teach a stochastic text transformer to reason. Why would anyone expect that approach to work? I would have thought the more obvious approach would be to couple it to some kind of symbolic logic engine. It might transform plain language statements into fragments conforming to a syntax which that engine could then parse deterministica…

I think you should drop the "stochastic text transformer" label you have probably heard applied, and instead think of them as neural networks that they are. Reason being that the term says absolutely zero about capabilities but creates a subjective 'reduction'. It's just a thought terminating cliché.

Let's for the sake of argument assume current LLM's are a mirage but in the future some new technology emerges that offers true intelligence and true reasoning. At the end of the day such a system will also input text and output text, and output will probably piece-meal as current LLM's (and humans) do. So voila: They are also "stochastic text transformers".

Yes LLM's were trained to predict next token. But clearly they are not just a small statistical table or whatever. Rather, it turns out that to be good at predicting the next token, after some point you need a lot of extra capabilities, so that's why they emerge during training. All the "next-token-prediction" is just a way abstract and erasing name of what is going on. A child learning how to write, fill in math lessons etc. is also learning 'next token prediction' from this vantage point. It says nothing about what goes on inside the brain of the child, or indeed inside the LLM. It is a confusion between interface and implementation. Behind the interface getNextToken(String prefix) may either be hiding a simple table or a 700 billion-size neural network or a 100 billion sized neuron human brain.

Re: Is chain-of-thought AI reasoning a mirage?

#183

Post by Terr_ »

Terr_ wrote: post #145

A regular "chatting" LLM is a document generator incrementally extending a story about a conversation between a human and a robot... And through that lens, I've been thinking "chain of thought" seems like basically the same thing but with a film noir styling-twist. The LLM is trained to include an additional layer of "unspoken" text in the document, a source of continuity which substitutes for how the LLM has no othe…

[P.S.] I realize I left off an important part: If the regular version did not already "reason"... Why would we ever expect this kind of tweak to change that, and bright forth real reasoning?

The core algorithm hasn't really changed, we're just changing the (hidden) document so that it's a different style with a greater density of clues, so that it can more-effectively bullshit [0] output humans won't notice and dislike.

[0] Creating something that "sounds good" without any particular awareness or care about truth or falsehood.

Re: Is chain-of-thought AI reasoning a mirage?

#184

Post by safety1st »

Night_Thastus wrote: post #171
safety1st wrote: post #87

Earlier quoted context omitted.

I'm pretty much a layperson in this field, but I don't understand why we're trying to teach a stochastic text transformer to reason. Why would anyone expect that approach to work? I would have thought the more obvious approach would be to couple it to some kind of symbolic logic engine. It might transform plain language statements into fragments conforming to a syntax which that engine could then parse deterministica…

Congratulations, you've said the quiet part out loud. Yes, the idea is fundamentally flawed. But there's so much hype and so many dollars to be made selling such services, everyone is either genuinely fooled or sticking their fingers in their ears and pretending not to notice.

Flawed or not I found a lot of the counterpoints people raised totally fascinating. The most significant point probably being _diyar's, that we've tried multiple approaches and LLMs are currently doing it better than the others. But the more philosophical stuff is fun too like the notion that logic and emotion are not actually separable within human cognitive processes, so why would you assume logic will end up off in its own little box in an artificial intelligence. I mean what a fascinating field of inquiry

Re: Is chain-of-thought AI reasoning a mirage?

#185

Post by zahlman »

> However, people keep asking the least interesting question possible

Am I the only one who finds this a really strange turn of phrase? If people continually feel compelled to do something out of curiousity, is that not the very definition of "interesting"?

Re: Is chain-of-thought AI reasoning a mirage?

#186

Post by d4rkn0d3z »

glial wrote: post #9
mentalgear wrote: post #5

> Whether AI reasoning is “real” reasoning or just a mirage can be an interesting question, but it is primarily a philosophical question. It depends on having a clear definition of what “real” reasoning is, exactly. It's pretty easy: causal reasoning. Causal, not statistic correlation only as LLM do, with or without "CoT".

Correct me if I'm wrong, I'm not sure it's so simple. LLMs are called causal models in the sense that earlier tokens "cause" later tokens, that is, later tokens are causally dependent on what the earlier tokens are. If you mean deterministic rather than probabilistic, even Pearl-style causal models are probabilistic. I think the author is circling around the idea that their idea of reasoning is to produce statements…

Two thoughts:

1) As far as I recall this program of formalizing mathematics fails unless you banish autoregression.

2) It is important to point out that a theorem in this context is not the same as a "Theorem" from mathematics. Production rules generate theorems that comply with rules and axioms of the formal system, ensuring that they could have meaning in that formal system. The meaning cannot justify the rules though, fortunately, most know to use the rules of logic so that we are not grunting beasts, incapable of conveying information.

I think the author wonders why theorems that don't seem to have meanings appear in the output of AI.

Re: Is chain-of-thought AI reasoning a mirage?

#187

Post by RainyDayTmrw »

Terr_ wrote: post #183
Terr_ wrote: post #145

A regular "chatting" LLM is a document generator incrementally extending a story about a conversation between a human and a robot... And through that lens, I've been thinking "chain of thought" seems like basically the same thing but with a film noir styling-twist. The LLM is trained to include an additional layer of "unspoken" text in the document, a source of continuity which substitutes for how the LLM has no othe…

[P.S.] I realize I left off an important part: If the regular version did not already "reason"... Why would we ever expect this kind of tweak to change that, and bright forth real reasoning? The core algorithm hasn't really changed, we're just changing the (hidden) document so that it's a different style with a greater density of clues, so that it can more-effectively bullshit [0] output humans won't notice and disli…

The steelman argument - not to say that I agree with it - is that this capability was present all along, but due to variability in the behavior of the model, adding certain context, such as chain-of-thought, exposes it.

Re: Is chain-of-thought AI reasoning a mirage?

#188

Post by viccis »

adastra22 wrote: post #124
viccis wrote: post #68

Earlier quoted context omitted.

>As far as I can tell “generating context” is exactly what human reasoning is too. This was the view of Hume (humans as bundles of experience who just collect information and make educated guesses for everything). Unfortunately, it leads to philosophical skepticism, in which you can't ground any knowledge absolutely, as it's all just justified by some knowledge you got from someone else, which also came from someone…

FYI this posts comes off as incredibly pretentious. You think we haven’t read the same philosophy? This isn’t about epistemology. We are talking about psychology. What does your brain do when we “reason things out”? Not “can we know anything anyway?” Or “what is the correlation between the map and the territory?” Nor anything like that. Just “what is your brain doing when you think you are reasoning?” And “is what an…

>FYI this posts comes off as incredibly pretentious. You think we haven’t read the same philosophy?

Rigorous language often comes across as pretentious to any layperson, especially when it concerns subjects like philosophy. I don't know what philosophy you've read, but, based on my experience, it's a pretty safe assumption that most AI practitioners do not own a well creased copy of Critique of Pure Reason.

>This isn’t about epistemology. We are talking about psychology. What does your brain do when we “reason things out”?

The only way to compare what our brain does (psychologically or neurologically) to what LLMs or other models do when we "reason things out" is via epistemology, which is to say "how is it possible to reason that out". Asking how our brains do it psychologically or neurologically is really not relevant, as LLMs are not designed the same as our brains.

>Philosophy doesn’t have answers for questions of applied psychology.

I think that expecting philosophy to have any "answers" for topics that include metaphysical questions is unreasonable, yes. But to even bring up "psychology" when discussing generative probability models is unhelpful anthropomorphization.

Re: Is chain-of-thought AI reasoning a mirage?

#190

Post by hakfoo »

gshulegaard wrote: post #101

> but we know that reasoning is an emergent capability! Do we though? There is widespread discussion and growing momentum of belief in this, but I have yet to see conclusive evidence of this. That is, in part, why the subject paper exists...it seeks to explore this question. I think the author's bias is bleeding fairly heavily into his analysis and conclusions: > Whether AI reasoning is “real” reasoning or just a mir…

Discerning the limits is the most important thing of all, and we seem very eager to obfuscate it for LLMs.

We so desperately want something we can sell as AGI or at least magic that the boundaries on the tools are few, far-between, and mostly based on legal needs "don't generate nudes of celebrities who can sue us" rather than grasped technical limits.

The more complex and sophisticated the query, the harder it will be to double-check and make sure you're still on the rails. So it's the responsibility of the people offering the tools to understand and define their limits before customers unknowningly push their legal-assistant LLMs into full Sovereign Citizen mode.

Post reply on HN