Superintelligence: The Idea That Eats Smart People (2016)

The problem is that the thesis cannot be disproved by one counter example (real or not), because the existential hazard comes from the possibility of just one such instance of an AI.

> it might be a defining feature of intelligence.

But if not, and the alarmists are right, deep shit. That's why the author's argument is void. The author needs to show that the chance of an AI with a simple goal and hyperintelligence is negligible. "it might be" is a coin toss, that doesn't help.

I offer a different, human-centric, POV: we are in a managerial-driven society with actual trend is more and more form Ford-model workers to use and control for business.

Human in this model are a problem: too stupid they are unavailing, not-that-much-stupid do not accept to be used as "the manager in charge" want. So? Better found replacements. Mechanical one. AI one. So the rest of human being can be used for reproduction, pleasure, gameplay, whatever and form them as stupid as possible is not a problem anymore since they are essentially unneeded for managements, only toys.

So today's managers dream "intelligent systems" in factory, "AI developers" (they already manage to cut off admins, too powerful and expert (they know the big picture so they can act autonomously) to being controlled as today devs), they dream a future in witch they simply order something to a machine and get obeyed without contestant, law limitation etc. TV series like Continuum, films like V for vendetta, The Handmaid's Tale only foreseen, as good artist do, that kind of future. Mass production of fantasy films with a religious background do the same on the opposite field.

That's the real troublesome point for us. AI today is not "AI", only pattern matching for automation without any comprehension of the physical world. New dictatorship instead is already there and more and more powerful.

Consider a thing: we do vast majority of our social critical tasks (identity management, banks operation, medicine, ...) on computers, so on proprietary systems made by very few more and more powerful vendors. Our data pass through very few more and more giants webservices, so someone else computers, free communication software was put into oblivion (from usenet to email to the concept of desktop-centric, personal computing from Xerox Alto&c to LispM passing through Plan9 and modern GNU/Linux desktop) substituted or jailed in proprietary systems (VoIP tech are mostly open but nearly all VoIP calls pass through very few companies, emails are open standard but most of user now are pushed to webmails etc). We are at a point of jail free software over a proprietary blackbox (think about WSL, DeX, Crostini), we have invented locked-down hardware that can only run proprietary software (secure-boot &c). Now imaging how easy can be steering the entire society for few people in the right position. The rest is simply a matter of time.

> The penultimate premise is if we create an artificial intelligence, whether it's an emulated human brain or a de novo piece of software, it will operate at time scales that are characteristic of electronic hardware (microseconds) rather than human brains (hours).

I never understood that argument, why would that be true? If I train a state of the art convnet it takes days on the most efficient hardware I can efford, why on earth would we go from nothing to superintelligent that also runs incredibly fast. Isn't it way more logical to assume that it will be slow at first?

Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years.

Malicious humans are in general easier to deal with since we know most of them are subject to ingrained psychological tendencies we have learned much about over the course of history and they depend on other humans to pull off something as catastrophic as a world war.

A general AI would have certain immediate advantages over humans:

  - Immensely larger communication bandwidth
  - Broader knowledge than any single group of humans
  - Faster thinking speed (at least by enlisting other processors)

Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you?