Superintelligence: The Idea That Eats Smart People (2016)

The serious statement he makes is that humans have complex motivations, and he thinks any superintelligence would as well--in fact it might be a defining feature of intelligence. The Rick and Morty reference was just a humorous example of what might happen, not an actual argument. The subheading was "the argument from complex motivations", not "the argument from Rick and Morty", so let's try and focus on the serious part and not the humorous aside.

At the scale of the brain we do.

what I find interesting is that for most of my life, Moore's law was... as real as gravity; It seemed like any problem you had, wait a few years and you could just brute force it.

The future really did look limitless.

Now, moore's law has slowed down. Hell, I've got a 7 year old laptop next to me, and it still works for games. I mean, in the '90s, a 7 year old computer would be a museum piece.

I mean, for big AI advances now, we're more and more dependent on software.

Really, though, I think this seems to be how most tech works. There are periods of rapid innovation where it looks like you are going to the moon. Like, even spaceflight. In the years from 1944-1969 we went from the first practical rocket that could lob a bomb across the channel (without much accuracy, as i understand it) to actually putting human beings on the fucking moon. The literal moon. I mean, 1961-1969 might be even more amazing; vostock 1; the USSR sent Gagarin into outer space, and the US decided they needed to one-up them; so the US went to the moon.

Aand... we kinda stopped after that. I mean, we sent out some probes and some robots, but like manned space flight has kinda petered out... and I still can't take a rocket if I want to go visit my buddies on the antipode and I don't want to spend all day in an airplane. Like, I can't buy a flight to Australia in under 12 hours for love or money, which probably is about what I'd say if it were 1969. (on the other hand, I can get said flight- round trip- for rather less than a day's pay for an Engineer, which is amazing, if you ask me.)

And a lot of that is physics. Like, in most technologies, there's areas where progress is slow and hard and areas where once you break through some hard part, it becomes easy to advance... for a while. but that easy streak never lasts forever. Not because people are dumber or smarter, but because exponential progressions, in nature, rarely stay exponential very long.

I guess I'm expressing doubt about the 'hard takeoff' just 'cause it seems to me like there is probably limits to how smart you can make a brain the same way that there are limits to how small you can make features on a silicon wafer.

I'm not saying you won't be able to make a machine that is way smarter than me (or even way smarter than any human) - I'm just saying there are almost certainly limits.

That's a really interesting thought. Essentially, the technical analogy of what a 'soul' could be.

Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances.

Personally, I find it amazing that humanity survived the cold war. the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future.

But, the fission bombs were expensive, centralized weapons. there was a small number of men who could have pushed the button.

The problems I see now aren't general AI; the problems I think you should fear is that we have very powerful tools that are very cheap. I mean, like the deepfakes thing. If instead of making creepy porn, what if people used that technology to cause chaos? Make targeted ads/videos of leaders saying nutty things? I mean, I think we could adapt to deepfakes style attacks fairly quickly; someone would have to use it decisively while it's new. But things like that are being invented all the time.