Machine intelligence, part 2

I commented above, but thought it wise to also comment directly to you. I enjoyed the musing of a Bane scenario, and consider it inside the realm of possibility. However, I'm of the opinion that SMI will have the ability to have 'constantly game-breaking thinking.' Which basically means there is no way we could possibly control it after a certain point. For instance what if It realizes how to move matter remotely?

I also wanted to discuss your point about the finite set of top AI researchers. I think their specific psychological biases have the potential to appear and magnify inside the personality of their AI, the way a work of art can be said to imbue the 'spirit' of its artist. Your thoughts would be appreciated.

Eliezer wrote something relevant to these types of proposals eight years ago: http://lesswrong.com/lw/jb/applause_lights/ when he asked someone calling for "democratic, multinational development of AI":

  Suppose that a group of democratic republics form a
  consortium to develop AI, and there's a lot of politicking 
  during the process—some interest groups have unusually
  large influence, others get shafted—in other words, the
  result looks just like the products of modern democracies.
  Alternatively, suppose a group of rebel nerds develops an
  AI in their basement, and instructs the AI to poll everyone
  in the world—dropping cellphones to anyone who doesn't have
  them—and do whatever the majority says.
  Which of these do you think is more "democratic", and
  would you feel safe with either?

A lot of these points sound nice: "we should regulate this and that" "we should be careful with such and such", but what would actually incentivize any politician ostensibly regulating AI to do the right thing? Or even find out what the right thing is?

This is a feel-good article, but all I can ever see it accomplishing is actual harm by advocating the idea that the worst decision-making process humanity has developed (politics) should be applied to the most dangerous threat humanity has to face (self-improving AI).

I don't mean to sound too harsh, it's nice to see more and more people beginning to take the threat AI poses seriously, but I think it's a dangerous leap to reach for the first tool in your toolbox (regulation) to try and handle such a problem.

Im confused as to the reference to SILEX.

Can someone elaborate?

In the year 3000, the machines were losing the war to humans. In a last act of desperation, they sent back a few of their own kind to the year 2015 to alter the course of history and skew the odds in the favor. All they needed to do was convince the readers of a tech news site that the notion of an evil AI was ludicrous.

They down-voted and and ridiculed the various stories prophesying the rise of AI, many which flew under the radar of those who had the greatest chance of halting the impending doom. By the time anyone realized what was happening, it was too late.

I think it's because Nick Bostrom's book Superintelligence is the first general reader overview of the subject, and it came out middle of last year.

Musk definitely read it, and I assume it's doing the rounds of the tech elite.

It is a good book, worth reading with plenty of references. It amusingly makes its own point - it tries to analyse what AIs might be and do and how to control them.

The analysis is such a mess, and shows our collective knowledge of this is such a mess, you can't help but agree with the author we need to put more attention to it.