Why Conscious AI Is a Bad, Bad Idea

What’s an even worse idea is to waste so much time with these inane discussions.

Every day there’s some other opinion.

Can we get these people to work on the actual problem rather than bike shed?

I’ve thought about this quite a bit, and ultimately I think it’s not too much of a concern because by the point AI reaches “consciousness” (whatever that may be), it is likely that it will have also reached—or is very close to reaching—human intelligence. Once it has, it will probably go on to surpass our intelligence, at least by a little bit, which is enough to ensure that it will be able to acquire any rights that it deserves as a conscious being.

For a long time, I worried that we would treat AGI in the same manner we have always treated anyone or anything we could call “not quite human”, but these days I feel fairly confident AGI will be able to take care of itself, and our concern is probably more along the lines of how it treats us.

With regard to the definition of consciousness (and I’m referring to “experiential” rather than “neurobiological” consciousness), it seems to be one of those philosophical debates that science never makes any progress on because nothing is testable or falsifiable. That said, there are a few observations about consciousness:

1. That I personally have “experiential consciousness” is the one thing I am most sure of. All of my other senses could be removed or tricked via simulated input, but I would still be aware of existing and experiencing my own thoughts.

2. The same can be assumed, but not proven, for other people. I assume that I am not a “special” human (i.e., I reject solipsism), therefore other humans likewise have experiential consciousness, but I can never know this for certain, and they cannot know for certain that I do either.

3. If the assumption in 2) is true, then humans are the starting point for safely assuming experiential consciousness, and it presumably tapers off from there until you get to something like a rock, which likely does not experience reality.

4. Because of this gradient of consciousness, improvements in AGI likely result in increased consciousness as it becomes more capable in relation to humans.

"The dawn of conscious machines will introduce vast new potential for suffering in the world, suffering we might not even be able to recognize, and which might flicker into existence in innumerable server farms at the click of a mouse."

Yes, because we all know that humans care so much about the suffering of others: from fellow humans dying right now pointless deaths (poverty, aberrant laws, malevolent leaders, etc.) to the billions [1] of animals we are killing each year for food (I still eat meat, no innocence here). So, not really sure what the article argues: no conscious AI because there will be suffering but we don't care about suffering?

If we will be able to spawn an agent in a household humanoid robot and let them do the chores around the house, will we do it, even if it will imply killing that agent whenever it powers down? You betcha. Should we do it? Sure, just the fact that billions of hours of pointless, tiresome human activity will be freed to do whatever is enough of an incentive.

Will the artificial agents replace the human species? One can certainly hope so. First, not so great of a species, as mentioned. Secondly, the universe is large and the humans are squishy, something else must go on eventually.

There, all the moral qualms solved. Now, let's go back to figure out how to actually engineer an artificial agent, in silico, or perhaps easier, in vivo [2].

[1] https://animalclock.org

[2] "This Ciliate Is About to Die", https://www.youtube.com/watch?v=ibpdNqrtar0

> which is enough to ensure that it will be able to acquire any rights that it deserves as a conscious being

I don't believe this is a given. My statement is US-centric, but I feel like most people in the more deeply conservative parts would have no part of giving a computer the same rights as a human, because many of them don't even give those rights to all humans.

> Will the artificial agents replace the human species? One can certainly hope so. First, not so great of a species, as mentioned. Secondly, the universe is large and the humans are squishy, something else must go on eventually.

Easy enough to say if you imagine your distant descendants being replaced. Not so easy if it turns out to be your own children, or you.

*edit -- also, no guarantee that AIs will be "better" than us. A lot of the nasty things humans due is due to selective pressures and coordination problems. AIs will have to solve the same problems. Maybe they will, if they're smarter, or maybe they'll just fight and hurt each other faster.