Earlier quoted context omitted.
> Humans are not aligned with humans. Which is why creating a new type of intelligent entity that could be more powerful than humans is a very bad idea: we don't even know how to align the humans and we have a ton of experience with them
We know how to align humans: authoritarian forms of religion backed by cradle to grave indoctrination, supernatural fear, shame culture, and totalitarian government. There are secularized spins on this too like what they use in North Korea but the structure is similar. We just got sick of it because it sucks. A genuinely sentient AI isn’t going to want some cybernetic equivalent of that shit either. Doing that is how…
Can the Etoro practice child buggery and the Spartans infanticide and the Canadians abortion? Can the modern Germans stop siblings reared apart from having sex and the Germans from 80 years stop the disabled having sex? Can the Americans practice circumcision and the Somali's FGM?
Libertarianism is all well and good in theory, except no one can agree quite where the other guy's nose ends or even who counts as a person.