There are potential parallels with how AIs are trained and the evolutionary pressures that our brains likely evolved under. Human AI trainers accept/reject or give a rating to the AI's response so the selection pressure on AI models is to produce a response that is likely to be accepted by the humans "in its environment" or at least to mirror the responses of humans it's seen provide in its training data.
Long ago I was struck by an idea I heard about human reasoning that I paraphrase as "human reasoning evolved not to reason, but to provide reasons"[1]. Basically that the heaviest evolutionary pressures on our brain were for social utility, like for influencing others to do something for us, or giving pre/post justifications for our actions that most others are likely to accept to avoid punishment. The flawed mechanisms that we developed to do such things are not significantly grounded in logical reasoning, but can sometimes be pressed into duty for that.
LLM models are trained to "generate the best next token" which augments the response so far and makes the evaluator happy. When I reflect on my own mind responding quickly to someone in a meeting it doesn't feel altogether different from that. We don't typically carefully and fully logically reason before we start talking. We typically start talking and think only a little ahead about what we say next that supports and doesn't contradict what we've said already. As we're speaking we are monitoring the other person's facial expressions to infer their emotional response and adjusting our next words based on that. I think at least common neurotypical[2] conversation might be closer to the "next token" reasoning than we would be comfortable admitting.
[1] I _think_ who I heard say that was Hugo Mercier, on either Sean Carroll's consistently amazing science podcast or maybe on Lex Friedman before he veered into Joe Rogan emulation. I'm normally a little suspicious of the theories of psychologists and cognitive scientists, but I intuitively related to many of his lines of thinking.
[2] I'm arguably pretty neurodivergent in more than one way so I think my thought processes are a little different by default, but that makes me more reflective about common conversation. When I'm trying to mask and communicate more similarly to a neurotypical corporate employee I feel like I become a "stochastic parrot" that's just predicting the best next few words to improve the emotional response I'm reading in my conversation partner.