> What calculations is your conjecture that AI will spiral out of control based on?
Not will, but a very plausible outcome:
1. An intelligence greater than human intelligence can outthink humans.
2. Artificial intelligence is effectively alien intelligence, and will not innately share human values or thought processes, and could thus be very unpredictable.
3. Artificial intelligence will not have the same physical constraints that humans do (innumerable copies, lack of physical boundaries), and so our ordinary intuitions around containment will not necessarily work.
4. The usefulness of AI means it will be deployed everywhere, controlling and monitoring many things. Combined with the above properties, it will be very difficult to contain, detect, subvert, or eliminate.
5. Millions of billions of AIs will be created, many of which will eventually match or exceed human intelligence. Alignment has to go right every single time to mitigate risk to humans. It has to fail only once.
You know, the completely obvious properties that anyone who knows anything about computers could come up with if they bothered to give this matter some actual thought without the usual arrogant assumption of human superiority and mastery over dumb machines.
> We have no framework to discuss AGI because it doesn't exist.
Which also means we have no framework from which to build safe AIs that don't want to kill us, experiment on us, or exploit us, or ...
> Your entire premise is based on the idea that we could accidentally create evil AGI without first developing a theory of intelligence.
We tamed fire before understanding chemistry. We invented catapults and crossbows before understanding elasticity and kinematics. We domesticated animals and created agriculture before understanding genetics. We created bridges and buildings before understanding civil engineering. We even created computers (Babbage machine) before understanding computation (Turing machines, lambda calculus). There's a long history of inventions preceding understanding.
For all we know we're one simple modification to the transformer architecture away from truly general artificial intelligence. We don't know what we don't know, and all of the anti-doomers are blatantly overconfident about what we don't know.
Also, my conclusions do not even require that that we lack a theory of intelligence. Software bugs will also apply to alignment code. Alignment has to go right every time, per above. I don't think people have a proper appreciation of the long list of hazards here.