Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario

I'd think they are quite independent.

Not sure I have any odds on those individually, other than I consider the overall risk really really low. The way I see it, there are a few things working against the risk from pure intelligence to start with (as it is with humans btw., the Nazis were not all intellectual giants, for example) and then it goes down from there.

International bans need treaties, with agreed standards for intervention in order to not escalate to war.

The argument is that if you're not willing to go all the way with enforcement, then you were never serious in the first place. Saying you won't go that far even if necessary to enforce the treaty is analogous to "sure murder is illegal and if you're formally accused we'll send armed cops to arrest you, but they can't shoot you if you resist because shooting you would be assault with a deadly weapon".

I'd reconsider your revision of your estimate of Yudkowsky, as you seem to be dropping it for not proposing the very ideas he spent the last 20+ years criticizing, by explaining in every possible way how this is a) the default that's going to happen, b) dumb, and c) suicidal.

From the way you put it just now:

- "develop an AI that is compliant to our wishes" --> in other words, solving the alignment problem. Yes, this is the whole goal - and the reason Yudkowsky is calling for a moratorium on AI research enforced by a serious international treaty[0]. We still have little to no clue how to approach solving the problem, so an AI arms race now has a serious chance of birthing an AGI, which without the alignment problem being already solved, means game over for everyone.

- "and can dominate all other AIs" --> short of building a self-improving AGI with ability to impose its will on other people (even if just in "enemy countries"), this will only fuel the AI arms race further. I can't see a version of this idea that ends better than just pressing the red button now, and burning the world in a nuclear fire.

- "actively sabotage competing efforts" --> ah yes, this is how you turn an arms race into a hot war.

> Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success.

Any limit on conventional weapons that had any effect was backed by threat of bombing the living shit out of the violators. Otherwise, nations ignore them until they find a better/more effective alternative, after which it costs nothing to comply.

Nuclear weapons are self-limiting. The first couple players locked the world in a MAD scenario, and now it's in everyone's best interest to not let anyone else have nukes. Also note that relevant treaties are, too, backed by threat of military intervention.

Cryptography - this one was a bit dumb from the start, and ended up barely enforced. But note that where it is, the "or else bombs" card is always to be seen nearby.

Can you see a pattern emerging? As I alluded to in the footnote [0] earlier, serious treaties always involve some form of "comply, or be bombed into compliance". Threat of war is always the final argument in international affairs, and you can tell how serious a treaty is by how directly it acknowledges that fact.

But the ultimate point being: any success in the examples you listed was achieved exactly in the way Eliezer is proposing governments to act now. In that line, you're literally agreeing with Yudkowsky!

> If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card.

There are fates worse than death.

Think of factory farms, of the worst kind. The animals there would be better off dead than suffering through the things being done to them. Too bad they don't have that option - in fact, we proactively mutilate them so they can't kill themselves or each others, on purpose or in accident.

> At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips.

With AI, there is no difference. The "use AI to dominate everyone else", besides sounding like a horrible dystopian future of the conventional kind, is just a tiny, tiny target to hit, next to a much larger target labeled "AI dominates everyone".

AI risk isn't like nuclear weapons. It doesn't allow for a stable MAD state. It's more like engineered high-potency bioweapons - they start as more scary than effective, and continued refining turns them straight into a doomsday device. Continuing to develop them further only increases the chance of a lab accident suddenly ending the world.

--

[0] - Yes, the "stop it, or else we bomb it to rubble" kind, because that is how international treaties look like when done by adults that care about the agreement being followed.

Out of curiosity how has that changed McKinsey's advice?