Superintelligence: The Idea That Eats Smart People (2016)

As engineers, we should work together against deep reinforcement learning degeneration for industrial applications. In that sense, if you have a blog or website or code repository to share for reference, please enlist it here to help create a critical mass. I am just trying to set up something useful at http://www.reinfle.com , hopefully it is going to help in the coming months.

Thanks for the link!

> If intelligence is fundamentally linked to specific sensorimotor modalities, a specific environment, a specific upbringing, and a specific problem to solve, [...]

> There are currently about seven million people with IQs higher than 150 — better cognitive ability than 99.9% of humanity — and mostly, these are not the people you read about in the news.

We measure IQ with pretty general pattern matching. An IQ of 150 is interesting, but those people are very very close to us. If we are talking about human specific tasks we should look at how IQ correlates with let's say playing poker, performing well on X factor, etc. IQ predicts how well someone will earn money in a very vague sense, but maybe there are better tests to know if someone will be able to persuade humans. (Do we have data about the IQ of fraudsters? Politicians? Bible/tupperware sales agents?)

> Intelligence expansion can only come from a co-evolution of the mind, its sensorimotor modalities, and its environment.

It might be true, but that's largely because humans can't self-modify their brains. And the fundamental problem is we can't even modify our minds at will. We can sure train ourselves, but we can't fundamentally alter our minds, because it's too tied to our brains.

> If the gears of your brain were the defining factor of your problem-solving ability, then those rare humans with IQs far outside the normal range of human intelligence would live lives far outside the scope of normal lives, would solve problems previously thought unsolvable, and would take over the world

Or simply 150 IQ is meaninglessly low. It's still in the human envelope. (Since we know there are 7 million people with IQ of 150.)

What happens around 300? Or around 500? That's so far out of the human range (depending on how the IQ test is calibrated, 200 IQ is one in ~4.8 billion for the 16SD one: https://www.iqcomparisonsite.com/iqtable.aspx )

These essays arguing against the possibility somehow forget that there are things we would benefit from, a lot. Simple faculties of the mind we are missing. Like a great alarm clock, a simple to-do list in your mind. (Sure you can train yourself to not forget things and wake up at specific times, but it's not like downloading a perfect alarm/todo app.) Working memory. We're at around 6 or 7. And it's very hard to push it up even by one in general. ( But it's possible to kind of try and train yourself to do this, which seems to help with working memory: https://www.gwern.net/DNB-FAQ#does-it-really-work )

Also, it's important to note, that that people who wanted to take over the world were usually very disturbed individuals. The human psyche is not really well suited to senseless killing while also perfectly simulating empathy and other things. (Of course psychopaths are a real thing.)

> Most of our intelligence is not in our brain, it is externalized as our civilization

And that's the point that should really scare people. Because we can't cram all that into our heads, but a human level AI, being software connected to a lot of storage, can.

That said, it's understandable that we can't understand the how of the intelligence explosion. That's basically the number one premise of the whole thing. But we can't simply disprove it. Any kind of proof that is so sloppy that simply throws around a formula like "because no complex real-world system can be modeled as `X(t + 1) = X(t) * a, a > 1`" can be ignored. Because we are not interested in a general complex system, we are interested in an AI that approximates that explosion. It might be linear with a very-very steep slope, or it can be just a nice parabola. We're still fucked.

> Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as bottlenecks.

This is a very strong point. That said, an AI can work around the bottlenecks. With self-modification nothing prevents it from reinventing itself from scratch every time it encounters a bottleneck. Diminishing returns are of course a problem for it.

> scientific progress, because it is conceptually very close to intelligence itself — science, as a problem-solving system, is very close to being a runaway superhuman AI.

Yes, and no. Because the components are still very much "just" humans. And that's the bottleneck. And we can't reinvent science from the ground up to replace that bottleneck. (Yet.)

> What bottlenecks [...] are slowing down recursive self-improvement [..]? So many, I can’t even count them. Here are a few [, that] would also apply to recursively self-improving AIs.

Let's see them!

> It gets increasingly harder to keep up with the firehose of new publications.

Yes, but no. This means if the AI can represent and access knowledge efficiently, then it'll be able to handle it, just like Google is perfectly capable of handling a growing World Wide Web.

> As scientific knowledge expands, the time and effort that have to be invested in education and training grows, and the field of inquiry of individual researchers gets increasingly narrow.

Because we can't do direct mind to mind knowledge transfer. Which is again something AIs don't really have as a constraint.

> conclusion

remember, if everything goes according to [my] plan, then we don't have to worry. so don't worry.

I mean, okay, sure, but WTF.

This seems wrong on its face: it conflates intelligence and consciousness (hence the various premises about brains being computable). You don't need a machine to be conscious for it to be intelligent, or superintelligent.

Changing your political regime gives you UBI. That's literally the only way to get UBI. It's not gonna happen magically by itself.

No, if there's no ground to do so. Chemtrailers believe that there are gigantic conspirancies to subdue humans (and ultimately worsen the condition of the whole humanity) but we shouldn't take them seriously.