Naah, I study cognition (besides a basic familiarity with the topics you mention) and even there, probabilities and bayesian models of cognition are well regarded by anyone familiar with mathematics. So, yeah, they are good. I think whenever you want optimal/rational inference under a closed set of alternatives, probabilities will do you good. But is the assmption of closed set of alternatives good? I doubt it. The set of all alternatives does not make sense, unless you specify a context, which almost everyone does when they are trying to model.
As much as I want to explore these other topics, I have not. So my understanding of these is limited to "they exist, but are underexplored, and it's not clear to me they can or cannot be reduced to probabilities". I hope to explore them in another life or another decade. I'm currently stuck on causality, but even here, they talk about normal and abnormal events. But what is the probability of me eating a cabbage today?
- https://direct.mit.edu/books/monograph/3540/Reasoning-about-...
- https://www.worldscientific.com/worldscibooks/10.1142/8665
- https://www.cambridge.org/core/books/an-introduction-to-nonc...
Understanding formal proofs and understanding natural language are as far apart to me as day and night.
I make no claims (or at least don't want to) about the superiority of human cognition (by what metric?). I do make claims about their similiary however. And they are not identical, which you seem to be claiming save for differing trainign data.
Let me make a distinction between values that arise by mere existence (hunger, thirst, fear of death, lust), and values we pick up as we grow up (religion and social norms). Machines can acquire the second, but acquiring the first requires being embodied in the world that is different from putting a microchip into a robot body. I don't do one job over the other because I was exposed to some training data that said I should do a job. But I do it because I value earning enough money to not starve myself to die, amongst a host of other values including wanting to enjoy what I do. Even the notion of enjoyment comes from embodied existence. You don't discover you enjoy something before you try it out. You don't always learn what you enjoy by looking at other people. Now, when you start giving AI the threat of death and the joy and pain of life, perhaps you can arrive at something similar to humans.
If someone started using mechanical help to the extent their healthy muscles and body deteriorated, that'd be a matter of concern too. For example, when you only drive and never walk. But this is exactly what is happening whenever you put AI in the hands of (unwilling) learners! I don't understand how using an LLM maps onto using medicines or prosthetics. They are clearly not the same! If you know a disability that LLMs can be used as a medicine or prosthetic for, please let me know!
Mainstream AI does not even understand causality except to parrot cases it has seen in the training data. There is a whole field of research in causal inference that needs to make its way to mainstream AI. So, good luck putting an AI that works solely on associations and correlations in charge of its own body let alone a nuclear reactor or the state.