"But how do we make a computer program that decides what it wants to do? How do we make a computer decide to care on its own about learning to drive a car? Or write a novel?" Perhaps it'd be better to ask: why would we want a computer to do these things? I certainly do not want to live in a world where computers have their own motivations and desires and the ability to act on the same. Actually, I can put that more s…
you seem pretty convinced living computers would kill us all. this seems like a pretty big assumption to me?
I assume that an AI will be more intelligent than us if we build it right. Then assuming that a randomly designed intelligence has the same goals as us is a huge assumption. Most humans don't even have the same goals, and we're 99% similar to each other.
In other words - the assumption that a random intelligence shares our goals is a much bigger assumption than that a random intelligence will be just like us.