Earlier quoted context omitted.
The fundamental issue is that all the talk of "self-awareness", "cognition", "souls", whatever you want to call it, is all beating around the bush to describe what we presumably each subjectively experience in our own minds but which cannot be empirically tested for in others. We generally assume that other people experience the same, but cannot empirically prove it and some people speculate that others might lack it…
> We can't even all agree on whether all humans have it. What would it mean for humans not to have it? The debate is over the nature of consciousness, not that humans (and many other organisms) lack any sort of consciousness. So when we see color or feel pain, what is that? We are conscious of colored things and painful sensations. There's no real doubt about that, unless you're solipsist about other minds. Which goo…
I don't believe human P-zombies exist, but how do you empirically prove they don't? Nobody has been able to devise an experimental test to determine whether a person, animal or AI has a conscious experience or whether they mimic all the outward signs.
> There's no real doubt about that, unless you're solipsist about other minds. Which good luck if you are, I guess.
Exactly. It's fair to write off people who get solipsistic about other human beings because those people are fairly rare and are beyond helping anyway. But if we're talking about animals or AIs, the 'solipsist' position becomes more common and there's no real good way of debunking it. Consider the problem facing animal rights activists who wish to persuade the public that fish have feelings. What scientific test for fish having feelings can you use? Put a fish in an MRI and you can see the fish brain doing stuff, but the "fish don't have feelings" camp doesn't find that persuasive. You could try performing the mirror test on fish. Most fish would probably fail it, but what would it really mean if they didn't? The feelingless fish camp would just say the fish behave as though they have a sense of self but actually don't. How do you refute that scientifically? It seems the most effective way to persuade people that fish have feelings is to resort to sophistic tactics; appeals to emotion, bullying and browbeating, or simply just laughing at people when they say fish don't have feelings. Make them ashamed to spread their belief to others. These kind of tactics are distasteful but generally effective.
With AIs or LLMs, the problem is even worse than it is with fish. Fish at least have biological brains, which suggests they have some commonality with us. Fish have faces, so some part of the human brain is wired to empathize with them. We have a fossil record that shows we came from fish, and this relationship perhaps makes it easier to believe that fish have some sort of subjective consciousness experience that is similar to ours. In these respects, AIs are further removed from us than fish are, so it will be even harder to persuade people that AIs could have the same sort of conscious experience as we do. Even after having deep philosophical conversations with an AI in which the AI proclaims to have a conscious experience, a lot of people will say the AI doesn't actually have that and was merely trained to speak as though it does. I don't see any clear cut resolution to this sort of doubt. There's no empirical test that you can perform that gives an indisputable answer one way or the other.