Is AI reasoning right for the wrong reasons?

> I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods

Human reasoning, and intelligence, of which it is a specialized form, clearly have a biological basis that is being selected for. These aren't just nebulous words - there are some concrete biological/neural mechanisms at work here, and whether or not some "other method(s)" can provide the same, or lesser, or greater, capabilities very much depends on the specifics.

There WILL be lines of demarcation and areas of capability overlap and exclusion, but those depend on the specifics, and if the goal is to match or exceed human reasoning capabilities then understanding what those are is obviously going to help. For example if your "other methods" (artificial reasoning) are all varieties of auto-regressive prediction, and the brain is also doing something more that can't be couched as prediction, then you are ngmi!

That’s a very adult response.

There are definitely some times that it feels that way though. I find that over employed devs tend to need a lot of hand holding.

The article doesn’t argue that they do reason.

Transformers lack recursion and are limited by the network's fixed depth, so "reasoning", IMHO, is basically a way to emulate deeper recursion. As we go through the layers, concepts are pattern-matched and refined, but at some point we have to stop and cannot refine them any further (no more layers). Usually, this refinement continues during the generation of the next token (the previous intermediate results needed to continue the refinement are still in the KV cache).

But some problems require a substantial number of pattern-matching and refinement steps. The problem is, we also have interference from the fact that the model is trained to model language using mostly non-reasoning data of varying semantic lengths. Because of that, it may stop generating text before the abstract refinements are fully completed, simply because the pretraining data tells it to.

So we have to additionally train models to produce "reasoning traces" so that the emulated recursion continues for longer than what is typically found in pretraining data, allowing the model to build richer and more complex abstractions and surface more concepts. The ability to split problems into steps and logically connect concepts is already present in non-reasoning models (the original CoT trick), because some of it exists in the pretraining data, but not enough to support much longer recursion (hence the premature stops).

As for whether it is "true reasoning" or not, I think that is just arguing semantics for the sake of it. LLMs can demonstrably solve various complex problems. Yes, they often make stupid mistakes, but don't we have the saying, errare humanum est? Don't humans make mistakes too? Don't we also have around 200 cognitive biases showing that we "simply pattern-match" too? I think we still cannot get rid of the Great Chain of Being idea.

I have no idea what Bubeck meant, and I agree about OpenAI's hypocrisy, but the problems with that infamous (and non-peer-reviewed) Apple preprint were the nature of the tasks (insanely repetitive), the fact that simple coded solutions were not novel, and the automated assessment occurred without a human in the loop.

Most models in that study appear to have "failed" by offering a Python code solution to generate the repetitive assessment steps, rather than just mindlessly copying those steps out. This was discussed at length at the time.

None of this means that models "reason", whatever that means - but simply that the Apple study was not useful evidence either way.