I don’t think it’s a foregone conclusion that human reasoning doesn’t operate on the basis of statistical prediction of the next most probable “token” at its most granular level. Humans are certainly capable of hallucination in the LLM sense, and without training we often struggle to produce (or even outright fabricate) the rationale behind our “conclusions”.
We stopped intentionally modelling human reasoning because we have no clue how it works.
We roughly copied the physical devices then figured out how to slap them together with an algorithm that makes them behave similarly to human thought, when trained with a massive quantity of cultural-linguistic-memetic data.
It should not be surprising that if you take a bucketful of engine parts and keep messing with them until they kinda work as an engine that you will probably wind up with something along similar lines as the original intention of the parts. Frankly, it would be quite surprising if you didn’t.
Biological neural networks are fundamentally linear algebra processing device that integrate data into functions through training, so the fact that we understand the process as linear algebra is actually an argument that the process is similar.
As to whether “superintelligence” can exist, I think we are in exactly the same page there. It is a matter of definition. Personally I doubt the existence of a fundamentally superior system of logic type of superintelligence, but certainly a device can made to iterate faster and flawless memory and instant, accurate calculation is going to beat the hell out of my notebook and HP28s lol.
I suspect that superintelligence can be (has been?) achieved in some respects, but in the same sense that a room full of ten educated people with regular computing tools is “superintelligence”.