Earlier quoted context omitted.
Weird that there’s so much resistance to the importance of figuring out (philosophically, practically) how to align AI to human wishes. The problem is already playing out right in front of our eyes, and that’s with humans in the loop every step of the way.
Isn't AI aligned to human wishes right now? Hateful rage-baiting news took over television/radio pre-AI. I think most humans are just evil and want extremism and division. Tribalism is what makes humans human. Outsiders need to be put down. Maybe the AI is preying on our primate brains, but it's what we want.
And to be sure, we don't understand this question well enough that it would be safe to commit humanity to our current ideas of the best solution for all eternity.
But we are certain that we wouldn't want a future AI governor of Earth to regularly and randomly pick someone to torture to death, just because that is what human drives have led to historically.