Superintelligence: The Idea That Eats Smart People (2016)

See, I always thought of the singularity, at least the hard takeoff as this idea that we'd come up with an ai that could make smarter copies of itself. Without any physical limits, that AI would get to the point where it was essentially infinitely smarter and infinitely more powerful than us.

I personally see that as fundamentally different from the difference between humans and the higher animals. I mean, that's a big difference, sure, but it's certainly finite.

I... don't think anyone is arguing that AI is safe in any way? I mean, the pattern matching tools we call AI now are already deployed in very dangerous weapons.

My point is not that AI is going to be safe and warm and fuzzy, or even that it won't be an existential threat. My point is that we're already facing several existential threats that don't require additional technological development to destroy society, and because the existing threats don't require postulating entirely new classes of compute machinery, we should probably focus on them first.

There are still enough nuclear weapons in the world for a global war to end society and most of us. A conflict like we saw in the first half of the 20th century using modern weapons would kill nearly all of us, and we are seeing a global resurgence of early 20th century political ideas. I think this is the biggest danger to the continuation of humanity right now.

We haven't killed ourselves yet... but the danger is still there. we still have apocalyptic weapons aimed and ready to go at a moment's notice. We still face the danger of those weapons becoming cheaper and easier to produce as we advance industrially.

This part however I can't stand behind:

> Anyway, let's just take this sentence: "We can measure the ability for systems to adapt creatively, as well as their resilience and their ability to use resources in an interesting way." - This is so broad and universal, that it's either meaningless or false. And there is absolutely no argument supporting this. I highlighted this, because this is what directly contradicts the alarmist thinking. If we were able to measure creativity and resilience in general, we could train AIs to get a higher creativity score, furthermore, we could then control them.

It's true that such generalist statements basically mean nothing. But your rebuttal doesn't take into account chaos theory -- where there's generally accepted that most living systems live on the brink of chaos yet are very stable and manage to swing back even after big interferences. Not sure what a "living system" is, don't ask. :D

I do agree with you that your highlight contradicts the alarmists though. Not everything swings out of control by the gentlest of touches. In fact, most of the universe doesn't seem to be that way. There's always a lot of critical mass that must be accumulated before a cataclysm-like event occurs.

---

All of the above said, I don't think it's serious or scientific to discard alarmists simply because the guys/girls with the most PR do ridiculuous or non-scientific statements. Behind them are probably thousands of people who are more systemic and have better arguments but aren't interviewed by mainstream media.

Where do I stand in the spectrum of this? The so-called singularity is possible. BUT, we are a very long way from it. We're going to be clawing our way to it, inch by inch, for centuries, if not millenia. That's what I think is most likely.

And by the time it occurs, IMO we will be living in a cyberpunk-like future -- very well articulated in the "Ghost in the Shell" anime movies and series by the way -- where the line between a man and machine will have already be blurred quite severely.