AI will not kill us, says Microsoft Research chief

This isn't a question of physical death it's a question of whether the human race has any place in the future other than as a historical footnote.

Just because the future will be very different doesn't mean we can't make good predictions about it (though the one you're referring to does seem like a bad prediction). For example we might predict that AGI will be interested in preserving itself and getting resources (http://en.wikipedia.org/wiki/Instrumental_convergence).

making Artificial General Intelligence would require us (or some machine) to clearly understand what intelligence actually is.

the correlation with processing power is I think mostly evangelized by Kurtzweil, who likes to point out that we can soon have a number of nodes in a neural network equivalent to the number of neurons in the human brain, or something along those lines.

but biologists have found secondary information encoding and replication systems which exist on top of DNA (sorry, it might take me hours to find this again, I should have bookmarked it). the minute we build a network which emulates a brain's network of neurons, we're probably just going to find a secondary layer within the neuron.

assume for the sake of argument that we eventually get through every layer of information process and can build brains in software, irrespective of whether it takes five years or five hundred years to get there.

there next needs to be a motivation for AGI to exist. general intelligence in human beings enables all conscious thought relevant to the existence of that human being. but all conscious thought relevant to the existence of a human being is an inherently different category of conscious thought than all conscious thought relevant to the existence of a machine or a piece of software, and to make any even remotely realistic guess about what that type of thought might be, you have to ask: what would be the meta-biological imperatives driving such a creature?

that question is very, very hand-wavy, so our answer would have to be very general, and my preference is also that an answer to such a hand-wavy question also be very cautious and diligent.

the one thing all machines have in common is that some living thing created them to serve its own purposes. it's reasonable to assume that any future machine we create will also be created for some purpose, and that any intelligence we imbue it with will exist to serve that purpose. it's absurd to assume that future technologies won't have unintended consequences, but it's also quite ridiculous to assume that some technology's unintended consequence will be a carbon copy of human consciousness.

the really irritating thing, also, is that corporations already exist, already have biological imperatives, already kill human beings, and already co-exist with human beings in a mostly peaceful manner, and already make it possible for our species to exist in much greater numbers, despite the fact that corporations also do very terrible things to specific subgroups from time to time.

so we already have an example of what probably is a new life form, comprised both of humans AND of machines.

and any new life form which achieves sentience will probably have those characteristics -- it will probably kill individual humans and specific subgroups, it will probably be beneficial overall for the species as a whole, and it will probably be made up of both machines and people.

all living things have self-preservation as a biological imperative.

so any new life form which comes into being is going to have that imperative also. so for any life form which comes into being as a result of our technological development, one major goal at all times will be preserving both the machines and the humans that is made of.

EDIT: you might get a heart transplant, but there's no good reason to replace all your mitochondria with something new.

The stakes with AGI are such that we can't really afford to be wrong.