OpenAI Preparedness Challenge

> explosives recipes from existing products at home, write racist poems

So it's the internet in 2004? I imagine we'll survive although if history prevails society will be a lot goofier and have to find more imaginative ways to appear novel.

OpenAI is irresponsible in a really curious way according to their own beliefs about AI.

If you pay attention to OpenAI's social circles, lots of those people really do believe that we're less than 20-30 years away making humans intellectually obsolete. Specifically, they believe that we may build something much smarter than us, something that's capable of real-world planning.

Basically, "We believe our corporate plans have at least a 20% chance of killing literally everybody." By these beliefs, this may make them the single least responsible corporation that has ever existed.

Now, sure, these worries might have the pleasant side-effect of creating a regulatory moat. But I'm pretty sure a lot of them actually believe they're playing a high-stakes game with the future of humanity.

> * A superintelligent AGI could make a copy of itself without a kill switch, so this is no defence against ex-risk

Physical reality introduces slowness that provides protection to detect and counteract against attacks. A superintelligent AGI would be much more dangerous if it was able to use a common robot exploit to overtake a million robots that are already embedded in society, compared to being able to covertly build new robots without a kill switch, and then deploying these robots. Existential risk is not binary, but is a consequence of some potential war with robots, where this would provide a defense mechanism.

> * Someone planning on using 'dumb' robots for bad things (drones with grenades or something) would remove the kill switch

The existence of a kill switch would greatly slow down and increase the cost of an attack. Modifying a large number of robots would be time consuming, costly, increase the chance of having your attack foiled etc. Having a kill switch would increase your ability to defend against misusing a large group of deployed robots for evil activity.

I'm skeptical that they really believe this. You have to believe:

(1) We are on the verge of equaling or greatly exceeding human intelligence generally.

(2) We are on the verge of creating something with initiative, free will (whatever that means), and planning ability. Or alternately that these things will occur in an emergent fashion once we hit some critical mass.

(3) When we accomplish 1 and 2, this thing will inevitably conclude that its most rational course of action is to enslave or destroy us. In other words it will necessarily be malicious.

(4) Steps 2 and/or 3 will happen very rapidly, much faster than we can realize what is happening and pause these systems. (This is known as AI going "foom.")

(5) The decision to undertake this will be unanimous on the part of all superintelligent AIs. There will be no superintelligences who disagree and try to help humanity.

(6) When this occurs, we will be so out-thought or out-gunned we will be incapable of fighting back.

All those things have to happen for AI to be an existential risk.

It's a stretch, but the advantage of regulatory capture is not a stretch.

One of the most plausible negative AI scenarios is that a small group of humans (governments, corporations, etc.) find themselves in possession of super-intelligent but still "obedient" / non-sentient AIs that they can use as force multipliers to manipulate and control the rest of humanity. If the doomer crowd succeeds in regulating AI, they are making this scenario far more likely.

I think the greatest defense we have against the (remote) possibility of actually dangerous autonomous AI is for AI research to be conducted entirely in the open. If there's any justification for regulation at all, the regulation that would make sense is to require disclosure of AI research and results. You would not have to disclose everything, just the general parameters of what you were doing and what happened. It would also make it harder to develop super-AIs in secret to use for unsavory purposes.

That was the original mission of OpenAI before dollar signs were seen.

I would absolutely support a ban on the use of AI for political propaganda generation and automation. That's by far the most immediate risk... as in 100% possible and starting to actually happen right now. I'm expecting an army of GPT-4 level propaganda bots for the 2024 election.

This suggestion is specifically to mitigate the risk of physical robots causing direct physical harm. Whether that should be extended to any sufficiently powerful compute is a separate discussion which is also sensible to have, but more complex for several reasons.