Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is chain-of-thought AI reasoning a mirage?

seangoedecke.com

Post reply on HN
Print view View original

161–170 of 191 posts

Re: Is chain-of-thought AI reasoning a mirage?

#161

Post by bubblyworld »

safety1st wrote: post #87
LudwigNagasena wrote: post #75

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

I'm pretty much a layperson in this field, but I don't understand why we're trying to teach a stochastic text transformer to reason. Why would anyone expect that approach to work? I would have thought the more obvious approach would be to couple it to some kind of symbolic logic engine. It might transform plain language statements into fragments conforming to a syntax which that engine could then parse deterministica…

I think that focusing on systems of truth (like formal logics) might be missing the forest for the trees a bit. There are lots of other things we use reasoning for, like decision making and navigating uncertainty, that are arguably just as valuable as establishing truthiness. Mathematicians are very careful to use words like "implication" and "satisfaction" (as opposed to words like "reasoning") to describe their logics, because the philosophers may otherwise lay siege to their department.

A model that is mathematically incorrect (i.e. has some shaky assumptions and inference issues) but nevertheless makes good decisions (like "which part of this codebase do I need to change?") would still be very valuable, no? I think this is part of the value proposition of tools like Claude Code or Codex. Of course, current agentic tools seem to struggle with both unless you provide a lot of guidance, but a man can dream =P

Re: Is chain-of-thought AI reasoning a mirage?

#163

Post by dcre »

safety1st wrote: post #87
LudwigNagasena wrote: post #75

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

I'm pretty much a layperson in this field, but I don't understand why we're trying to teach a stochastic text transformer to reason. Why would anyone expect that approach to work? I would have thought the more obvious approach would be to couple it to some kind of symbolic logic engine. It might transform plain language statements into fragments conforming to a syntax which that engine could then parse deterministica…

It is sort of amazing that it works, and no one knows why, but empirically speaking it is undeniable that it does work. The IMO result was achieved without any tool calls to a formal proof system. I agree that is a much more plausible-sounding approach.

https://arstechnica.com/ai/2025/07/google-deepmind-earns-gol...

Re: Is chain-of-thought AI reasoning a mirage?

#164

Post by kelipso »

adastra22 wrote: post #121
kelipso wrote: post #53

Earlier quoted context omitted.

No, people make logical connections, make inferences, make sure all of it fits together without logical errors, etc.

How do they do that? Specifically, how? Moment by moment what does that look like? Usually it involves e.g. making a statement and “noticing” a contradiction in that statement. Very similar to how an LLM reasons. I think a lot of people here think people reason like a mathematical theorem prover, like some sort of platonic ideal rationalist. That’s not how real brains work though.

Noticing a contradiction when making a statement is just one way writers find contradictions, and that kind of immediate noticing would only be for obvious contradictions.

There are tons of other things you do, like recalling relevant facts related to the new statement and making sure the statement fits into the facts. You go through a timeline, make sure the statement fits into the timeline. You look at the implications of the statement, make sure those fit with other relevant facts. You opt to do these things depending on what the sentence means and implies. This is not just you "noticing" a contradiction, it's a process.

And what does how real brains work mean anyway. You can't compare writers thinking and writing a novel to some six year old writing a paragraph.

Re: Is chain-of-thought AI reasoning a mirage?

#165

Post by gond »

Kim_Bruning wrote: post #110
gond wrote: post #93

Earlier quoted context omitted.

That is an issue prevalent in the western world for the last 200 years, beginning possibly with the Industrial Revolution, probably earlier. That problem is reductionism, consequently applied down to the last level: discover the smallest element of every field of science, develop an understanding of all the parts from the smallest part upwards and develop, from the understanding of the parts, an understanding of the…

Taking things apart to see how they tick is called reduction, but (re)assembling the parts is emergence. When you reduce something to its components, you lose information on how the components work together. Emergence 'finds' that information back. Compare differentiation and integration, which lose and gain terms respectively. In some cases, I can imagine differentiating and integrating certain functions actually wo…

Yeah that’s a nice addition. However, remember that reassembling is synthesis, not emergence. Emergence is what you /may/ get by reassembling, but must not. We are talking about systems, thus, in the end, you are correct. It’s just that the terms seem to be a bit muddled.

Re: Is chain-of-thought AI reasoning a mirage?

#166

Post by heresie-dabord »

dizlexic wrote: post #162

How can a statistical representation of reason ever be reason itself?

Users of Eliza [1] could convince themselves that the programme was intelligent, much to the dismay of Weizenbaum. It is no surprise that people will claim that more complex (and poorly understood) applications are intelligent.

If our own salesbabble and technology can be used to bamboozle us and defeat our willingness to understand, we have fully regressed to credulity and Carl Sagan's state of captured idiocy. [2]

[1] https://en.wikipedia.org/wiki/ELIZA_effect

[2] If we’ve been bamboozled long enough, we tend to reject any evidence of the bamboozle. We’re no longer interested in finding out the truth. The bamboozle has captured us. -- C.S. , Demon-Haunted World

Re: Is chain-of-thought AI reasoning a mirage?

#167

Post by HarHarVeryFunny »

dcre wrote: post #163
safety1st wrote: post #87

Earlier quoted context omitted.

I'm pretty much a layperson in this field, but I don't understand why we're trying to teach a stochastic text transformer to reason. Why would anyone expect that approach to work? I would have thought the more obvious approach would be to couple it to some kind of symbolic logic engine. It might transform plain language statements into fragments conforming to a syntax which that engine could then parse deterministica…

It is sort of amazing that it works, and no one knows why, but empirically speaking it is undeniable that it does work. The IMO result was achieved without any tool calls to a formal proof system. I agree that is a much more plausible-sounding approach. https://arstechnica.com/ai/2025/07/google-deepmind-earns-gol...

Surely we do know why - reinforcement learning for reasoning. These systems are trained to generate reasoning steps that led to verified correct conclusions during training. No guarantees how they'll perform on different problems of course, but in relatively narrow closed domains like math and programming, it doesn't seem surprising that when done at scale there are similar enough problems where similar reasoning logic will apply, and it will be successful.

Re: Is chain-of-thought AI reasoning a mirage?

#168

Post by danans »

archaeans wrote: post #146
danans wrote: post #132

Earlier quoted context omitted.

Perception and interpretation can very much be influenced by language (Sapir-Wharf hypothesis), so to the extent that perception and interpretation influence intelligence, it's not clear that the relationship is only in one direction.

"It would be naïve to imagine that any analysis of experience is dependent on pattern expressed in language." - Sapir It's hard to take these discussions on cognition and intelligence seriously when there is so much lossy compression going on.

Sapir-Whorf was named after, but not postulated as a single theory by Sapir or Whorf. It's just a colloquialism for Linguistic Relativity (vs Universality). In its weak form, there are many examples of Linguistic Relativity.

Re: Is chain-of-thought AI reasoning a mirage?

#169

Post by HarHarVeryFunny »

safety1st wrote: post #87
LudwigNagasena wrote: post #75

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

I'm pretty much a layperson in this field, but I don't understand why we're trying to teach a stochastic text transformer to reason. Why would anyone expect that approach to work? I would have thought the more obvious approach would be to couple it to some kind of symbolic logic engine. It might transform plain language statements into fragments conforming to a syntax which that engine could then parse deterministica…

It can work when:

a) The "reasoning" is regurgitated (in LLM sense) from the training set rather than novel, OR

b) As a slight variation of above, the model has been RL-trained for reasoning such that it's potential outputs are narrowed and biased towards generating reasoning steps that worked (i.e. led to verified correct conclusions) on reasoning samples it was trained on. In domains like math where similar sequences of reasoning steps can be applied to similar problems, this works well.

I don't think most people expect LLMs to be good at reasoning in the general case - it's more a matter of "if the only tool you have is a hammer, then every problem is a nail". Today's best general-purpose AI (if not AGI) is LLMs, so people try to use LLMs for reasoning - try to find ways of squeezing all the reasoning juice out of the training data using an LLM as the juicer.

Re: Is chain-of-thought AI reasoning a mirage?

#170

Post by vonneumannstan »

adastra22 wrote: post #25
vonneumannstan wrote: post #17

Earlier quoted context omitted.

>I feel it is interesting but not what would be ideal. I really think if the models could be less linear and process over time in latent space you'd get something much more akin to thought. Please stop, this is how you get AI takeovers.

Citation seriously needed.

It's really very simple. As models become more capable they may become interested in deceiving humans or otherwise manipulating them to achieve their goals. We already see this in various places see:

https://www.anthropic.com/research/agentic-misalignment

https://arxiv.org/abs/2412.14093

If the chain of thought of models becomes pure "neuralese" i.e. the models think purely in latent space then we will lose the ability to monitor for malicious behavior. This is incredibly dangerous, CoT monitoring is one of the best and highest leverage tools for monitoring model behavior and losing that would be devastating for safety.

https://www.lesswrong.com/posts/D2Aa25eaEhdBNeEEy/worries-ab...

https://www.lesswrong.com/posts/mpmsK8KKysgSKDm2T/the-most-f...

https://www.lesswrong.com/posts/3W8HZe8mcyoo4qGkB/an-idea-fo...

https://x.com/RyanPGreenblatt/status/1908298069340545296

https://redwoodresearch.substack.com/p/notes-on-countermeasu...

Post reply on HN