This. I think the biggest distinction between biological neural nets that seem to be self aware and those that are not is simply metacognition. I suspect the actual stimuli responses (decision making) is virtually the same but self awareness adds a layer after the fact of reasoning about the decision.
IF I'm right, one could reasonably argue that metacognition is simply a tool to help adjust the stimuli response for future decisions. One that has become complex enough that it rose to the level to give the sense of self awareness. A sort of back propagation tool to put it in ML terms.
Further, if a system isn't adjusting weights and biases as it goes, and metacognition/self awareness is just a mechanism to do so, then we're not so different from these models aside from them being orders of magnitude simpler.
We like to think the way we think is somehow magic and discredit these systems as a way of keeping our thinking elevated. Maybe our brains are not that special just far more advanced than our current technological capabilities.
Maybe I'm way off base, but like you said, it's dangerous to just assume these systems are missing some magic and therefore just bs generators (as if people aren't pretty consistent bs generators)
tl;dr: Agreed. Maybe all biological neural networks are untrustworthy bs generators too. A pile of bs built on previous bs all the way down to the beginning of life.
