Is AI reasoning right for the wrong reasons?

Ah, so you're more in the Investor mindset than the Scientist mindset. All you care about is results, not how it got there. There's a whiff of "hey, it's magic!" to that.

LLMs are one-trick pony's - they use the past to predict the future (presumed to be the same as what they were trained on). i.e. they are trained as auto-regressive predictors.

LLMs learn two slightly different types of reasoning via two different types of training.

1) SFT, or even base model training, on data that contains reasoning traces, learnt via next token error feedback. This does not result in "stochastic parroting" in the naive/pejorative sense, but nonetheless is very context dependent, even if the usual generative multi-source mashups apply.

2) RLVR post-training, where the model learns to mimic long-horizon (not just next token) reasoning via boosting a sequence of next-token predictions that steer the output towards a verified reasoning step (i.e. one that was at least valid in the context of the RL training sample). As Karpathy has noted, this is a pretty crude mechanism since you reinforce everything - errors included - that lead to the verified outcome.

RLVR is more powerful than SFT, and can result in more generalizable reasoning, since it is operating at a higher level of entire long-horizon reasoning steps, and also critically because it is most successfully being applied in the domains of math and coding which are highly self-consistent and logical. A reasoning step that was valid in one context should be equally valid in another context as long as you have successfully learnt what that generalized context is. Therefore, in these domains, you can chain together sequences of individually learnt reasoning steps, and hopefully this "novel" assembled reasoning chain is valid as a whole.

So, what is still missing from LLM reasoning compared to human reasoning? No doubt humans reason by memory a lot of the time too, and reductive axiomatic math reasoning works just as well for humans as when automated. So, what's missing?

There seem to be two major things.

1) RLVR requires rewards, and how well it works is going to depend on how accurate those rewards are. Is this reasoning step actually valid, or does it just kinda look ok? When moving beyond the cold reductionist logic of math and coding, the notion of correctness is far weaker, and it seems the best you can do is train on human curated reasoning rubrics and LLM-as-judge, which is much more fallible, leaving the model really needing (but lacking) a fallback to more general reasoning, not just memorized "maybe correct" reasoning steps.

2) Whether for reasoning outside of math & coding, or even within these domains when hoping for super-human innovative reasoning, not just lego-assembly proofs, what LLMs are lacking is a mechanism for what to do when next token/next step prediction fails. What LLMs currently do is "hallucinate", not even recognizing the failure.

In the human brain 50% or more of our cortex is feedback paths and the machinery that (perhaps together with the archaic part of our brain) lets us recognize and respond to failed predictions in an adaptive manner. This starts with continual learning (prediction failure being the signal), but also includes critical innate traits such as curiosity, boredom and frustration, that provide impasse resolution by encouraging us to explore unknown environments/contexts, abandon exploration when it is not productive, and generally expose ourselves to learning situations.

The dream is for AI scientists making new discoveries - the AI that could have invented general relativity if it has lived in Einstein's time, but this is not going to happen until their reasoning stops being purely predictive and becomes creative as well - curious about their own knowledge gaps and pursuing them in directed fashion, etc.

The current crop of Erdos solutions etc, while useful, really just represent the "generative closure" of what can be done/discovered WITHOUT learning anything fundamentally new. These will no doubt continue for a while until the more exhaustive search supported by computers has found the majority of these unexplored paths, and then we will need to move beyond LLMs to more brain-like architectures and algorithms that have the capacity for real innovation and discovery.

The alternative is to posit that you know the True Definition of thinking, which is kind of absurd.

Some things, like scientific laws, are outside of us (well, to a first order approximation - but generally I agree with this), but a concept such as "thinking" is pretty clearly going to be very wishy washy and subjective and changing with the times.