Why Conscious AI Is a Bad, Bad Idea

I'm not convinced of this a priori. And even if it was proven to me, the non-self-aware impulse-to-live would still be enough for me to find it morally significant. It would feel morally significant to me euthanizing a human neonate with a lethal condition that would otherwise prevent it from ever getting to a self-aware stage of life.

That's just false. Do you think humans are some God chosen species which makes them magically matter while the rest (including aliens on other planets I suppose) lacks that godly spark?

> I am having trouble deciding if a conversation with you is possible, you do not seem to be a rational being. You use the word "bad" in a way that another person would if they were saying "this is either bad for me personally, for those I care about, or humanity in general".

Yeah, and any normal human would understand the sentences "my cat hurt its foot, I hope it doesn't suffer too much." and "If it suffers, that would be bad".

> Does that mean you'd include sea urchins and wombats in "humanity in general"? If it did, then you're mentally ill in a way I'm not qualified to treat.

Who said sea urchins are humans?

> When a lion eats a gazelle in Africa (or anywhere else, I suppose, if such events occur), it is nothing that I or other humans should care about. It does not matter that the lion eats it while it is alive instead of "slaughtering it humanely" (imagine how I have to word that idea, it's absurd).

Well, humans can do little about suffering of animals in the wild, but that doesn't mean their suffering doesn't matter. Your own pain doesn't matter less when it can't be treated.

> If I could push a button to increase the intensity of that "suffering" a thousandfold, or increase its quantity (or even both!), is there any reason not to push that button? If I didn't push the button, mind you, it would only be because the suffering of those gazelles matters not to me one way or the other. If I could decrease the intensity/quantity with a different button, I wouldn't push it either... and for the same reason.

This seriously sounds psychopathic.

I’m not trying to catch you in some kind of “gotcha” but trying to understand your reasoning. If the moral implications of killing animals are roughly the same as killing plants, do you also believe agriculture needs reform for similar reasons? And if not, wouldn’t that indicate some higher moral obligation towards animals?

> Without self-awareness, there is no 'someone' to reflect on experiences. No personhood.

Why is reflection on experience the bar and not experience itself? Animals demonstrate learned behaviors, e.g. recognizing humans from memory and resuming friendly behavior based on that recognition. Similarly, avoidance of situations that are known to cause pain.

Our not-so-distant primate ancestors had a similar kind of experiential existence before gaining the ability to reflect on that experience.

The underlying experiences that this reflection reveals are the same experiences that predate our ability to self reflect and are the parts of us that are most common to us and other animals.

I guess what I’m fundamentally not understanding in your argument is the basis for the idea that a species gaining self reflection somehow becomes the point at which it becomes immoral to kill or harm that species.

Furthermore, moral behavior can be found all throughout the animal world, with clear indications of love/protection, companionship, sharing/cooperation, reciprocity, memory of transgressors, etc. Obviously the subjective experience of these behaviors will differ across species, but the more important point is that it seems problematic to attribute the existence of moral behavior to self awareness. Self awareness helps us improve our understanding of moral behavior through rational thought, but the logic of such inquiry ultimately still relies on those underlying subjective experiential states. The fact that through introspection we can identify and label these concepts is unique to humans, but what I conclude about this is quite a bit different than your claim.

I’d argue that gaining the ability to self reflect is the very thing that increases our moral obligations. Only through self reflection can we realize that as a species, we’re no longer bound to our evolutionary defaults, and no longer required to kill other animals to survive. What arguably started as natural selection of traits that are adaptive (but imperfect) for the survival of a social species could evolve beyond those more primitive defaults. And through self reflection we can now understand what pain feels like, and how inflicting it on others is harmful - to them and to us.

I’m not arguing that eating animals is never acceptable. But the way we go about it surely seems to matter, and if it matters, the implications of it mattering are worth exploring more broadly.