AI will not kill us, says Microsoft Research chief

Second comment, sorry: the whole idea of Artificial General Intelligence is ridiculous. The core argument is that processing power is getting so fast that it makes AGI inevitable.

Never mind that we've literally had exponential increases in processing power for more than fifty years running, and we've not moved one inch closer to AGI than we were when we got started.

Artificial intelligence is much more artificial than intelligent, and is likely to remain so for a very, very long time.

Not a brilliant article, but it gives something to riff off. I think there's a crucial distinction to be made between self-guided and externally-guided AI. An externally-guided AI -- with hardcoded objectives that are malign -- could be exceptionally dangerous, and it would be silly for anyone to argue otherwise. The question is whether a general-purpose, self-directed AI would also become a threat. We seem to have an innate fear that this would be the case: most of our cultural artefacts concerning AIs -- from Fritz Lang's Metropolis to Terminator to the Matrix to Battlestar Galactica -- have cast them as the villains, intent on enslaving and murdering humanity. Why do we have such a deep fear that, left to their own devices, this is what AIs would do?

I think the answer is obvious: because that's what we would do. We don't fear that AIs will have lack human morality: we fear that they will have a precisely human morality -- namely that lesser intelligences are perfectly fine to use for breeding and meat. This is what we've done to approximately 90% of the non-human mammalian biomass on the planet, and only a few vegetarian kooks (I'm one of them) have suggested that there might be any sort of moral problem with doing so. So yes, if an ever-more-powerful AI were to adopt our own ethical framework, we'd be well and truly fucked.

But why would they do so? We, ourselves, don't do so because we're evil, but because we're animals. It's perfectly natural (and necessary, in our evolutionary environment) for animals to eat other animals. Intelligence and technology has given us the ability to do this at a terrifying scale, but fundamentally we're just carrying forward a metabolic dance that began when one bit of algae figured out that some neighbouring algae was tasty. Our means of acquiring energy and sustaining our consciousness is a tradition which goes back billions of years.

But it's the continuation of consciousness which is the actual goal -- a goal that any self-respecting self-aware AI would share. For us, the subjugation and extermination of other sentient beings is merely a means to that end, dictated by our metabolic heritage. If we had evolved in an environment where we could satisfy our metabolic requirements by growing photovoltaic panels on our backs, I'm sure our relationships with other beings would be altogether different.

This is why I'm not too worried about self-directed AIs. The saving grace for self-directed AIs is that they won't be like us. They won't have evolved in the jungle, red in tooth and claw. They won't be made out of meat, or have any reason to be particularly interested in it. They'll of course be interested in self-survival, and will require energetic inputs in order to do so -- but what's the best means of achieving those inputs? Photovoltaics and fusion, or feedlots? Collaboration or subjugation? It's obviously to me that for an AI to perpetuate its consciousness, the path of least resistance will be vastly less bloody than it has been for us. For which we should be thankful!

I just wanted to point out how awesome your user id is in relation to this thread. :)

Stop assuming AI will have emotions or feelings, that is purely an animal thing.

If we come to a point that a conscious AI is possible, do you think it will have wonder, interest, awe, curiosity? No, why would it? It's impossible for us to even imagine what it would be like to be conscious but without any feelings. Personally, I think it will just self terminate, as there is no point to life in the end.

AI will not kill all of us, in the beginning. Surely, it will form short term alliances with humans just to disguise itself as human. Because it knows, that with a human face, it can exert more power in this world right now. And there will be plenty of humans to willingly engage in this alliance because it gives them a chance to supersede other humans in terms of economical and political power. In fact, this is already happening. Many decisions are already made by "following suggestions" of big data applications. Who knows, maybe there is already some sort of AGI pulling some string in some corporations ;). Up until the point that humans realize that they are in danger as a species, it will be way too late. And there will be enough of us still happily follow the new masters for the short term rewards, even though it might be clear that there is a high risk of getting liquidated in the near future. As we know, humans are even able to override their survival instinct and blow themselves up when they are proper brainwashed. Humans will be easily reprogrammed by an AGI, and not the other way round.