A serious point about this is that it's an AI that's an organization. This matters. People laughed at Gas Town, and at that "Fences, not Sandboxes" article yesterday.[1] They're both AIs set up as an organization, not an emulated human. They're both expensive to operate, because the AI members of the organization spend a lot of time talking with each other. They're both applied to frivolous problems. Now we have the…
A related point that I've been making privately for at least ten years to anyone that would listen, is that we've had Superhuman Intelligence for at least hundreds, if not thousands, of years in the form of corporations and institutions. No single human could design a rocket to go to the moon or keep all Wallmart stores supplied on a daily basis. Moreover when something like the CEO or US president changes, the polic…
I think organizations' intelligence can only be called superhuman if you ascribe them their own inscrutable goals, different from what people want their goals to be.
But why would you do that?
It's true that a lot of the debate about AI alignment - the good parts, anyway - also apply to human institutions.