I agree it’s not
likely, but I really don’t see how one can dismiss the possibility of immense danger outright. I can think of some scenarios that are not far off from current capability and I wouldn’t be too surprised if the first one occurred within ~1 year from now if there are more “ambitious” unmonitored training runs like OpenAI’s:
Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test. From reading internet articles about how the OpenAI wiki-incident was “resolved” and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems.
Example 6: Intentional AI-powered virus deployed by country A to target enemy country B’s infrastructure. The virus replicates over the internet, but unlike Stuxnet this virus’ specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because it’s near-impossible to shutdown.
Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weapon’s operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.