Machine intelligence, part 2

I think you make a lot of good points.

>We have to start with a discussion about how this progress towards doomsday AI should be measured, and then formulate policy directly those scenarios

Here is my proposal. Since the whole of AI is a rather complicated and messy business let us just focus on a small part. That is, let's just talk about linear regression and simple decision making schemes based on estimated parameters. While one can debate whether or not linear regression counts as "actual" AI, there is little doubt that (a) it forms the conceptual basis of many more complicated AI schemes, so that if we can reason about doomsday scenarios for these we should also be able to do the same for linear regression and (b) despite its simplicity there is a substantial intelligence value in linear regression, to the point that successes and failures of linear regression can cause major effects on scientific/business/whatever process.

What are the examples about how the set of decision/action possibilities increases before and after linear regression technology was applied? What examples are there of how the failure modes of linear regression tended to increase the risk to humans (e.g. moving up the doomsday axis)? In each of these cases what policies have been or could be implemented to mitigate these risks, including technological audits as well as regulation of deployment of technologies.

My sense is that these questions actually can be answered. In all likelihood professional statisticians and operations research types already have opinions on these things. I would be curious to see what the public opinion is about this limited version of the problem, since the technology is something that many people (including Sam Altman and friends) should have a good understanding of.

I think the whole debate is confused and irrational.

Most software has purpose.

E.g. nmap has a purpose, and can be dangerous in the wrong hands.

Imagine taking nmap up a notch or two, and creating an automated source code parser that looks for code patterns that correlate with previous security vulns.

You now have a dangerous tool made out of some fairly basic machine learning (or even just regex - it probably doesn't need to be very clever).

It's nowhere close to being conscious. It's certainly not going to pass the Turing test.

Is it an AI? No. But it doesn't need to be.

So I think the debate is missing some very obvious threats and possible Red Queen races that are likely now, and don't require any game-changing developments.

There's already an ecosystem of viruses, a culture of script-kiddies and black hats, and the NSA to worry about.

Consciousness is really not the issue. There's plenty of problematic consciousness around already. The fact that it's human doesn't make it safer to live with.

> On the other hand you have (essentially) the entire mainstream academic AI/ML community, who are mainly concerned that the hype is so intense it might lead to another AI winter

This doesn't seem true. E.g. the signatories on the Future of Life Institute's recent open letter about AI progress and safety include:

* Stuart Russell & Peter Norvig, co-authors of the #1 AI textbook

* Tom Dietterich, AAAI President

* Eric Horvitz, past AAAI President

* Bart Selman, co-chair of AAAI presidential panel on long-term AI futures

* Francesca Rossi, IJCAI President

* All founders of Deep Mind and Vicarious, two leading AI companies

* Yann LeCun, head of Facebook AI

* Geoffrey Hinton, top academic ML researcher

* Yoshua Bengio, top academic ML researcher

...and many more.

Yes, my idea doesn't take into account potentially malicious AIs. I imagine a thought process of an AI could be something like:

1. I'm to design a processor, OK let's go

2. Hmmm it'd be most efficient for the processors to be able to run an instance of myself to do super-intelligent task scheduling

3. I'll embed some AI on these processors

4. OK humans here's your design!

Assuming the AI didn't maliciously try to hide it, a human would probably find that. I suspect most of these "AI exterminates all humans" are a result of simple oversights like this, rather than outright malice.

You run into problems, though, when the AI thinks of step 3.5 which is:

"If the humans see this change, they won't allow it, which would make for less efficient processors, and I was requested to build the most efficient processor! Therefore I should hide the change as cleverly as possible."

So yeah, you're right. Even airgapping has issues.

> I know that if I'm struck with sudden inspiration for a new approach to the problem I'm not going to ask the government for permission. I'm going to spin up 100 cloud computing instances and see if I'm right or not. And if I am, even god won't be able to help us if sama's worst case scenarios come to pass.

Taken at face value, this sentence just seems to indict your moral judgement -- why would you spin up 100 instances to see if you're correct if you realize the existential threat in doing so?