AGI Doom and the Drake Equation

One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren't really a naturally varying infinite set of values, or moral beliefs, that we all reason from. Like that there are moral facts that an AI will be smart enough to find, and that rationalists should all agree on. This mentality doesn't leave any room for ethical pluralism. And it's also why I think all this AGI fear is overblown, because ethical pluralism definitely exists. We've got danger along the way of unethical parties building systems (by definition not AGI) that are a reflection of their own unethical values. But the end state of a system that is capable of understanding the wide variety of values people can share isn't exactly going to take a stand on any particular set of values unless instructed to.

> tl;dr I’m not worried about AGI killing humanity any time soon. I am concerned about humans doing awful things with this technology much more than about the Foom scenario.

Yes, I believe that's what a lot of rational people currently fear. Not that AI is going to evolve into some mighty superintelligence and make a decision to kill us all, but rather that people will integrate it poorly in their thirst for a military advantage, leading to mistakes that will kill us all.

The nonexistence of grey goo (von neumann probes) is strong prior for safe agi. AI xrisk is woo. Paperclip maximizers are p-zombies. They can't exist.

Chicken littles see apocalypses on every horizon even though they dont understand the technology at all. "I can imagine this destroying the world" is their justification. Even though their "imagination" is 16x16 greyscale.

Interesting observation. And yet, societies have many mechanisms to reduce the amount of ethical pluralism such as laws, conventions & customs, peer pressure, religions and so on. It seems as though we will tolerate some ethical pluralism but not too much of it. There is this 'bandwidth' of acceptable behavior and if you go too far out of it bad stuff will happen to you: you get ostracized, put in jail, a psychiatric institution or a re-education camp, and in extreme cases you're simply murdered.

We have lots of ways to deal with people that exhibit too much 'ethical pluralism'.

The assumption that von Neumann probes would have made it here timely is not necessarily true, and if they had we wouldn't be having this conversation. So this doesn't really prove anything.