Illah Reza Nourbakhsh's 2015 Foreign Affairs article -- "The Coming Robot Dystopia: All Too Inhuman" -- has an excellent take this topic [1].
All of the examples of AI Risk on safe.ai [2] are reasonable concerns. Companies should be thinking about the functional safety of their AI products. Governments should be continuously evaluating the societal impact of products coming to market.
But most of these are not existential risks. This matters because thinking of these as existential risks entails interventions that are not likely to be effective at preventing the much more probable scenario: thousands of small train wrecks caused by the failure (or intended function!) of otherwise unexceptional software systems.
Let's strong-man the case for AI Existential Risk and consider the most compelling example on safe.ai: autonomous weapons.
Nuclear weapons attached to an automated retaliation system pose an obvious existential risk. Let's not do that. But the "automated retaliation system" in that scenario is a total red herring. It's not the primary source of the threat and it is not a new concern! Existing frameworks for safety and arms control are the right starting point. It's a nuclear weapons existential risk with some AI components glued on, not the other way around.
In terms of new risks enabled by recent advances in AI and robotics, I am much more worried about the combination of already available commodity hardware, open source software, and semi-automatic weapons. All three of which are readily available to every adult (in the US). The amount of harm that can be done by a single disturbed individual is much higher than it has been in the past, and I think it's only a matter of time before the first AI-enabled simultaneous multi-location mass shooting happens in the US. The potential for home-grown domestic terrorism using these technologies is sobering and concerning, particularly in light of recent attacks on substations and the general level of domestic tension.
These two risks -- one existential, the other not -- entail very different policy approaches. In the credible versions of the existential threat, AI isn't really playing a serious role. In the credible versions of the non-existential threat, nothing we might do to address "existential AI risk" seems like it'd be particularly relevant stopping a steady stream of train wrecks. The safe.ai website's focus on automated cyber attacks is odd. This is exactly the sort of odd long-tail scenario you need if you want to focus on existential risk instead of much more probable but non-existential train wrecks.
And that's the strong-arm case. The other examples of AI risk are even more concerning in terms of non-existential risk and have even less credible existential risk scenarios.
So, I don't get it. There are lots of credible threats posed by unscrupulous use of AI systems and by deployment of shoddy AI systems. Why the obsession with wild-eyed "existential risks" instead of boring old safety engineering?
Meta: we teach the "probability * magnitude" framework to children in 6th-11th grades. The model is easy to understand, easy to explain, and easy to apply. But at that level of abstraction, it's a pedagogical toy for introducing children to policy analysis.
[1] https://www.foreignaffairs.com/coming-robot-dystopia
[2] https://www.safe.ai/ai-risk