Superintelligence: The Idea That Eats Smart People

While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you grant the AI risk crowd's assumptions. Elided from https://alexcbecker.net/blog.html#against-ai-risk:

The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that its masters start thinking about the next group, and the next. It's smart factories that create a vast chasm between a new, tiny Hyperclass and the destitute masses... AI is hardly the only technology powerful enough to turn dangerous people into existential threats. We already have nuclear weapons, which like almost everything else are always getting cheaper to produce. Income inequality is already rising at a breathtaking pace. The internet has given birth to history's most powerful surveillance system and tools of propaganda.

"I live in California, which has the highest poverty rate in the United States, even though it's home to Silicon Valley. I see my rich industry doing nothing to improve the lives of everyday people and indigent people around us."

This is trivially false. Over a hundred billionaires have now pledged to donate the majority of their wealth, and the list includes many tech people like Bill Gates, Larry Ellison, Mark Zuckerberg, Elon Musk, Dustin Moskovitz, Pierre Omidyar, Gordon Moore, Tim Cook, Vinod Khosla, etc, etc.

https://en.wikipedia.org/wiki/The_Giving_Pledge

Google has a specific page for its charity efforts in the Bay Area: https://www.google.org/local-giving/bay-area/

This only includes purely non-profit activity; it doesn't count how eg. cellphones, a for-profit industry, have dramatically improved the lives of the poor.

The laws of the universe forbid steam locomotives in space, and yet we reached mars regardless. And here we are, barely even regular Inteligences.

The laws of computational complexity seem like a stronger candidate for a hard limit, but that only means that to get smarter they will require more matter to distribute comoutations across.

The battery limit proposal is just silly.

Wooly Definitions: The only definition that matters is ability to make correct predictions about the future. All other things called "intelligence" are a consequence of this. This definition is formalized with AIXI, which is uncomputable, but computable approximations exist.

Hawking's Cat, Einstein's Cat: Scams require some intelligence from the victim. The victim needs to mistakenly believe they're doing something smart. Cats are too stupid to scam. Unlike humans they can't talk and they fail the mirror test, suggesting no self awareness. Human behavior when confronted with a deceptive superintelligence is not going to be the same as cat behavior when confronted with a deceptive human.

Emus: The name "Great Emu War" is a joke. The humans had limited resources available, because killing emus wasn't important enough to justify more. If we really wanted to kill all the emus we could do it. We've made plenty of other animals extinct. The motivation for an AI is set by its reward function, which can be as high as necessary.

Slavic Pessimism: This argument suggest that building nuclear weapons is impossible.

Complex Motivations: This isn't obvious nonsense, but consider that all intelligences we've seen so far are the result of evolution, which tends to produce a more complexity than needed. A leg is more complicated than a wheel. A non-evolved intelligence would not necessarily have complex motivations.

Actual AI: Compare with the argument from actual primitive nuclear reactors, which get mildly warm and never explode.

My Roommate: As a human, you roommate had a human reward function. Unsurprisingly, he acted like a human. Why should a non-human reward function result in human-like behavior?

Brain Surgery: Brain surgery would be a lot easier if you could take backups of brains, and duplicate them, and you had no concern about killing the patient.

Childhood: If this turns out to be needed, why can't increasing intelligence result in increased ability to simulate an environment suitable for raising superintelligences?

Gilligan's Island: There's no reason to assume an AI would be isolated. It could have or gain access to the internet and most human knowledge, and the mind architecture could contain many independent sub-minds.

Grandiosity: This depends on assigning ethical value to hypothetical humans, which isn't obviously correct.

Outside Argument, Megalomania, Comic Book Ethics, String Theory For Programmers: Ad-hominem.

Transhuman Voodoo, Data Hunger, AI Cosplay: Why should something be false because it's deplorable? And why should something be false because it encourages deplorable behavior?

Religion 2.0: Ted Chiang talked about the definition of "magic" in an interview (https://medium.com/learning-for-life/stories-of-ted-chiangs-...).

>Another way to think about these two depictions is to ask whether the universe of the story recognizes the existence of persons. I think magic is an indication that the universe recognizes certain people as individuals, as having special properties as an individual, whereas a story in which turning lead into gold is an industrial process is describing a completely impersonal universe.

All religions require some element of magic. Even Buddhism, which is arguably the least magical of all religions, treats consciousness as magic. AI requires no magic, therefore it is not a religion.

Simulation Fever: Simulated universes do not have to be magical by Chiang's definition. A universe could be simulated by something that pays no attention to individuals within the simulation, eg. something that lets a large number of universes run their course then examines them statistically. Possibility of this increases the possibility of living in a non-magical universe despite possibility of living in a simulation.

Incentivizing Crazy: This isn't an argument, it's a description of a field. Perhaps the author meant it to be an ad-hominem: "the idea is false because crazy people believe it".

Not that I take the whole Bostrom superintelligence argument too seriously, but this is an incredibly weak argument (or more accurately, bundle of barely-related arguments thrown at a wall in the hope that some stick) against it. Feel free to skip the long digression about how nerds who think technology can make meaningful changes in a relatively short amount of time are presumptuous megalomaniacs whose ideas can safely be dismissed without consideration, it's nothing that hasn't been said before.