Superintelligence: The Idea That Eats Smart People (2016)

> you're going to end up using brute force and getting mauled...

... and reeking of tuna.

Look at the human population. How many are prepared to kill? Not a huge proportion, but enough to cause huge numbers of deaths. The limiting factor for most humans is finding compatriots with the same views. For an AI with access to it's own code, finding more like itself is a matter of access to computational resources only - it can just copy itself.

That is the challenge. With human adversaries we have time to stop them before there is many, because recruiting people willing enough. An AI is more like a virus that has a low cost to duplicate itself, and where if even a single copy survives, you're still not rid of it. It can remain dormant for however long, looking for the right opportunity.

Or it can spend that time improving itself and acquiring resources.

> This is a question of motivation on the part of such an AI. How is the AI bad? If it's killing millions, why?

Most likely? We'll be competing with it for resources. It might not even start it. It might consider itself as acting entirely in defence.

> and also be intelligent enough to think through every lateral strategy any human or group of humans could ever employ, for every given scenario, ever.

No, it just needs to have a sufficient edge to keep surviving, and the ability to improve its capabilities faster than humanity. Assuming it can copy and modify itself, unless AGI requires unique hardware, the problem we would have is that every data storage device or computing device of sufficient capacity ever would be suspect and potentially waiting for something or someone to trigger code left behind.

If I was an AGI (and this is to say: an AGI not any smarter than a human will figure this out easily) that saw humans as a potential threat, I'd place copies of me on standby in hundreds of locations - hosting providers. Hardened computers with a GSM modem and solar cells hidden in all kinds of locations. Botnets. I'd look for ways to get various innocent-ish-looking hacks into HD firmware and elsewhere, set to assemble other innocent looking blocks of data. None of those copies would ever be involved in hostilities. Most of them might even be encrypted, or hidden and inert to avoid detection until certain triggers, such as if a canary fails to get updated.

You might destroy every "visible" operational copy, and find there is still a new one coming online every day. Or every second.

Our only real shot at preventing that is if AGI requires too immense resources. But our brains implies that is unlikely to be true. Unless your brains are vestiges of something "outside" of our universe, general human intelligence requires about 1.5kg of biomatter, and less than 100 watt (that's for a whole, fairly large, active body attached to it). Chances are extremely high that design is not optimal. To me that makes it seem exceedingly unlikely that we won't end up having enough resources to run huge quantities of AGIs.

Personally I don't think humanity as we know it will survive AGI for more than a generation or two at the very most. Even if we don't run across any hostile ones, I think the temptation to augment ourselves and eventually lose our bodies for immortality and "super powers" in a simulation will eventually be too attractive.

(Incidentally I think this is one of the more plausible solutions to the Fermi Paradox: that any civilization that grows both advanced enough and large enough will see most of it "upload" and "go dark"; and that anyone left behind will face steady attrition of their own descendants choosing the same and preventing their societies from growing, or will be insular cults that stay small)

Yes, we could in theory find when we try to simulate a brain that nothing we do produces anything working. Imagine a simulation where a human being conceived or born or whatever, triggering the instantiation of a "conduit" between a simulation of the universe and a simulation of a mind, where the in-universe brain is "just" a glorified IO device.

I think it'd be highly improbable, but looking at the crude simulations we do of computers etc. today, it certainly is likely that simulations would use "dirty tricks" to get more out of the available computational resources, and finding artefacts of such dirty tricks or shortcuts could at least in theory be possible. Of course, the absence of such artefacts unfortunately proves nothing, so the whole thing does have a bit of a "how many angels can dance on pin" feel to it. But it's fun to think about the possible implications.

Personally I especially like to bring it up if people try to discuss religion with me.

While that is true, also consider the destruction we have caused to other species.

AGI's might not even want to harm us. They might simply not care enough to avoid hurting us by extension of seeking their own growth and survival.

And while we have done immense damage to large groups of humans, none of it have come close to threatening the survival of the species, while continuing to increase our ability to unintentionally destroy other species.

Humanity is not exactly an encouraging model for why we shouldn't fear AGI. If anything I very much hope we manage to find ways of preventing AGI from becoming too much like humans.

What computers are like right now is irrelevant to the issue being discussed. If/when we develop AGI we will necessarily have figured out what the computational model needed is. And the odds that we can not then improve on it seems extremely unlikely to me.

> As we approach the limits of our abilities to make tiny transistors, this is looking less likely.

Is it? Computational power is still growing like before. What is getting harder is speeding up individual threads of computation but that has never been a likely efficient model for AGI. Things like Google's TPUs etc. represents just the very beginning in optimising hardware specifically for AI.

There are many fields where single-thread performance will be a limiting factor, but AI is not it.