Thoughts on OpenAI, reinforcement learning, and killer robots

I think the "If society can get through these issues, then..." perspective is fundamentally flawed. We will always, always have urgent problems. Guanranteed. We need to work hard on them. But we also need to work a little bit on the big things that are not of most immediate concern. I think a great analogy for this is climate change. If people in the early Industrial Revolution knew what we know now about greenhouse gases, they wouldn't have been inclined to worry. The amount they were putting into the atmosphere just was not significant, and even if it rose sharply, it would be many generations before there might be an issue. And yet, if people had put some focus on long-term threats, we might be in less dire straits now.

I definitely sympathize with your experience as an AI educator being constantly peppered with hypotheticals about superintelligence. I love your fast.ai course and I am sure you'd rather talk about teaching AI than about something that may or may not be an issue decades from now. But that doesn't mean the issue doesn't deserve more funding.

Musk's topic choice may seem repetitive to us, as we are in the tech world and read about/discuss this topic all the time. But a bunch of older politician types probably have not even been exposed to these ideas of existential risk, and as they hold a lot of power, it's not a bad thing to educate them on. For what it's worth, Musk also spent a lot of time discussing nearer-term implications like job displacement.

So why do black hat hackers exist? Why do you have so much faith in humanity's ability and/or willingness to "evolve" alongside AI? My prediction is that AI will evolve and humans will still be the same murderous creatures they are today, only with more tools at their disposal.

> You're also ignoring basic ideas that can defeat the apocalyptic problems you suggest. For example, "fake news" in the sense that you propose (adversaries generating synthesized videos and audio indistinguishable from reality) can be trivially defeated by cryptographic signatures.

How trivial is it for a political figure who is already untrusted by the public to assert that "cryptographic signatures" are a proof of fowl play? Again, we are talking about propaganda which plays off of human perception, not truth (the "post-truth world" meme comes to mind). DKIM signatures were often cited during the 2016 election by the alt-right as "proof" that the Podesta emails released by WikiLeaks were legitimate. The assertion that the content of the emails could have still been modified without invalidating the signatures wasn't enough to convince voters who continue to push conspiracy theories surrounding them. Humans are pretty easy to manipulate. What makes you think AI is incapable of doing this?

> You also ignore that one already has the capability to ruin a specific person's life via digital means today, without AI, just with enough dedicated time and money.

I acknowledge that. My fear is that with greater capabilities, AI could be used to deceive humans easier and on a greater scale. A murderer can kill someone with a baseball bat. They can kill hundreds of people with an AR-15. There's a reason why we have regulations on the latter.

The axiom I am starting from is not to demonize AI. I am not a luddite and recognize that AI has just as much potential to be used for good. AI is just another technology. My fear speaks more to how humans will use that technology as it becomes more sophisticated.

Put another way, consider the old "What is AI" joke: 'When the computer wakes up and asks, "What's in it for me?"'

---

edit: In case it's not clear, I agree with _qbjt, if for different reasons. The only people more frightening than the happy-go-lucky AI people are the happy-go-lucky synthetic biology people. Makes me glad I don't have children. It's all one big experiment now, with no control group.

Climate change and nuclear war are huge issues, and we correctly invest orders of magnitude more money and time (which might still not be enough) trying to prevent or limit these than trying to prevent the birth of a malicious AGI, but they are extremely unlikely to lead to a complete extinction of the human race.

Regarding the other scenarios, if you allow me to move the goalposts from pure risk assessment:

- Preparing against an alien invasion seems futile, since given the timescales at play in the universe, the first aliens we meet will likely be millions of years behind or ahead of us.

- One way to survive an asteroid impact is to colonize another planet ahead of time, which Musk is working on.

- Physics experiments requiring specialized hardware like the LHC already have much more oversight than AI research where a breakthough could potentially happen in a garage on commodity hardware.

So it makes a lot of sense to me to invest some money in preventing an AI doomsday scenario next.