Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is chain-of-thought AI reasoning a mirage?

seangoedecke.com

Post reply on HN
Print view View original

131–140 of 191 posts

Re: Is chain-of-thought AI reasoning a mirage?

#131

Post by handoflixue »

js8 wrote: post #98

I think LLM's chain of thought is reasoning. When trained, LLM sees lot of examples like "All men are mortal. Socrates is a man." followed by "Therefore, Socrates is mortal.". This causes the transformer to learn rule "All A are B. C is A." is often followed by "Therefore, C is B." And so it can apply this logical rule, predictively. (I have converted the example from latent space to human language for clarity.) Unfo…

Humans are also notoriously bad at this, so we have plenty of evidence that this lack of consistency does indeed cause failures on larger problems.

Re: Is chain-of-thought AI reasoning a mirage?

#132

Post by danans »

mrspuratic wrote: post #100
azrazalea_debt wrote: post #37

Earlier quoted context omitted.

A lot of current LLM work is basically emergent behavior. They use a really simple core algorithm and scale it up, and interesting things happen. You can read some of anthropic's recent papers to see some of this, like: They didn't expect LLMs could "lookahead" when writing poetry. However, when they actually went in and watched what was happening (there's details on how this "watching" works on their blog/in their s…

On that topic, it seems backwards to me: intelligence is not emergent behaviour of language, rather the opposite.

Perception and interpretation can very much be influenced by language (Sapir-Wharf hypothesis), so to the extent that perception and interpretation influence intelligence, it's not clear that the relationship is only in one direction.

Re: Is chain-of-thought AI reasoning a mirage?

#133

Post by safety1st »

_diyar wrote: post #114
safety1st wrote: post #87

Earlier quoted context omitted.

I'm pretty much a layperson in this field, but I don't understand why we're trying to teach a stochastic text transformer to reason. Why would anyone expect that approach to work? I would have thought the more obvious approach would be to couple it to some kind of symbolic logic engine. It might transform plain language statements into fragments conforming to a syntax which that engine could then parse deterministica…

We expect this approach to work because it's currently the best working approach. Nothing else comes close. Using symbolic language is a good idea in theory, but in practice it doesn't scale as well as auto-regression + RL. The IMO results of DeepMind illustrate this well: In 2024, they solved it using AlphaProof and AlphaGeometry, using the Lean language as a formal symbolic logic[1]. In 2025 they performed better a…

Very interesting stuff, thanks!

Re: Is chain-of-thought AI reasoning a mirage?

#134

Post by js8 »

handoflixue wrote: post #131
js8 wrote: post #98

I think LLM's chain of thought is reasoning. When trained, LLM sees lot of examples like "All men are mortal. Socrates is a man." followed by "Therefore, Socrates is mortal.". This causes the transformer to learn rule "All A are B. C is A." is often followed by "Therefore, C is B." And so it can apply this logical rule, predictively. (I have converted the example from latent space to human language for clarity.) Unfo…

Humans are also notoriously bad at this, so we have plenty of evidence that this lack of consistency does indeed cause failures on larger problems.

Yes, humans fail at this, that's why we need technology tnat doesn't simply emulate humans, but tries to be more reliable than us.

Re: Is chain-of-thought AI reasoning a mirage?

#136

Post by kromem »

LudwigNagasena wrote: post #75

> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "…

Latent space reasoners are a thing, and honestly we're probably already seeing emergent latent space reasoners starting to end up embedded into the weights as new models train on extensive reasoning synthetics.

If Othello-GPT can build a board in latent space given just the moves, can an exponentially larger transformer build a reasoner in their latent space given a significant number of traces?

Re: Is chain-of-thought AI reasoning a mirage?

#137

Post by mirekrusin »

brunokim wrote: post #69

I'm unconvinced by the article criticism's, given they also employ their feels and few citations. > I appreciate that research has to be done on small models, but we know that reasoning is an emergent capability! (...) Even if you grant that what they’re measuring is reasoning, I am profoundly unconvinced that their results will generalize to a 1B, 10B or 100B model. A fundamental part of applied research is simplify…

Feels like running psychology experiment with fruit flies because it's cheaper and extrapolating results to humans because it's almost the same thing but smaller.

I'm sorry but the only hallucination here is that of the authors here. Does it really need to be said again that interesting results happen when you scale up only?

This whole effort would be interesting if they did and plotted result while scaling something up.

Re: Is chain-of-thought AI reasoning a mirage?

#138

Post by mdp2021 »

adastra22 wrote: post #127
slashdave wrote: post #48

Earlier quoted context omitted.

Perhaps we can find some objective means to decide, rather than go with what "feels" correct

That’s not how our brains work though, or how must examples of human reasoning play out. When asking “do LLMs reason” we are asking whether the action being performed is similar to regular humans, not some platonic ideal of a scientist/rationalist.

> When asking “do LLMs reason” we are asking whether the action being performed is similar to

Very certainly not. We ask if the system achieves the goal.

"When we ask if the coprocessor performs floating point arithmetic, we ask if the system achieves the goal (of getting accurate results)". Not, "does the co-processor ask if we have a spare napkin".

Re: Is chain-of-thought AI reasoning a mirage?

#139

Post by mirekrusin »

okasaki wrote: post #28
stonemetal12 wrote: post #24

Earlier quoted context omitted.

> it was tested on a 4 layer deep toy model How do you see that impacting the results? It is the same algorithm just on a smaller scale. I would assume a 4 layer model would not be very good, but does reasoning improve it? Is there a reason scale would impact the use of reasoning?

Human babies are the same algorithm as adults.

This analogy would mean very large model that didn't finish training yet.

Tiny model like this is more like doing study on fruit flies and extrapolating results to humans.

Re: Is chain-of-thought AI reasoning a mirage?

#140

Post by mdp2021 »

adastra22 wrote: post #126
mdp2021 wrote: post #46

Earlier quoted context omitted.

But a big point here becomes whether the generated "context" then receives proper processing.

What processing? When you have an internal line of thought, what processing do you do on it? For me, it feels like I say something, and in saying it, and putting it into words, I have a feeling about whether it is true and supported or not. A qualitative gauge of its correctness. A lot of my reasoning is done this way, trusting that these feelings are based off of a lifetime experience of accumulated facts and the an…

I meant the LLM. I meant, with «whether the generated "context" then receives proper processing», whether the CoT generated by the LLM and here framed as further "context", regardless here of how properly it is generated, receives adequate processing by the internals of the NN.

A good context (any good context) does not necessarily lead to a good output in LLMs. (It does necessarily lead to a better output, but not necessarily a satisfying, proper, decent, consequential one.)

Post reply on HN