Is AI reasoning right for the wrong reasons?

First, we reason every millisecond on an ongoing basis which then can alter slightly or greatly with enviromental feedback. LLMs are turn based and token by token. Second its pretty unlikely that the token is the base element of our cognition, we created language far after we could do basic reasoning (advanced reasoning ala the greeks thats more debatable).

Theres a ton of research on the differences here, but I think its akin to this: we reason instinctually at an extremely high order level with super undefined "grains or vectors" that point to a wide variety of "objects or concepts or feature spaces". LLMs reason on one thing, token weights.

Sort of like the difference between pixels and reality. Pixels can represent reality, but they certainly are very very very flat and low resolution renderer of them, not reality itself. Even a 4K moving image is a flat redition of reality at best with only a tiny sample of the true experience. Media theory here can take over on the differences and the effects on humanity when they mistake one for the other.

1) LLM's written CoT might not always be faithful to the model's real reasoning process (true and important)

2) The "stochastic parrot" hypothesis, which the article reintroduces as "approximate retrieval" - ie, LLMs don't "really reason" at all, they just memorize a lossy encoding of their training data. This obviously raises the question of how LLMs can now routinely solve open mathematical problems, with no solutions in the training data by definition. The article handwaves this with:

"The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified."

The problem is that "mimicking" training data to arrive at a "plausible" result gets you an incorrect-but-plausible-sounding "proof" of the Jacobian conjecture, which was famous for humans writing plausible-looking "proofs" that had subtle flaws. You can't disprove the conjecture through sheer luck (search space too large) or "approximate retrieval" (the only thing you'd retrieve are fake "proofs"; far more human effort went into proof than disproof) or by writing something "plausible" that just happens to be correct (Jacobian was famous for "plausible" but wrong); the model must be carrying out mathematical reasoning somehow, by any sane definition of the word, even if it isn't fully reflected in CoT. The article doesn't address this.

It's clear that there's an enormous amount of leverage built into language-as-practiced that one can use to engage in a broad spectrum of reasoning, from the extremely fallible off-the-cuff conclusion to the deeply-considered and rigorous proof. How do we know this leverage is built into language-as-practiced? Because LLMs can do a broad swath of it.

But how do we know we're not doing something similar?

I don't think we can assume that we're not simply by observing that we're not digital and we don't use matrix multiplication. Why immediately dismiss the possibility that there might be a similar, but biomechanical, computation at play in our heads that plays in the same space of vectors?