Superintelligence: The Idea That Eats Smart People • Hacker News Board

Re: Superintelligence: The Idea That Eats Smart People

#561

Post by sweetjesus » Fri, Dec 23, 2016, 11:55 PM UTC

gbhn wrote: post #145

singlow wrote: post #131

Earlier quoted context omitted.

Their might is asymmetrical, but power is mitigated by the willingness of an organization of humans to follow commands. There is a limit to how far a soldier will go, ethically.

True, true, but somehow that's never been much of an obstacle to totalitarian governments. Somehow there's always a soldier who will push the button. In Nuremberg we developed a way to think about this: people outside the central circles of power have a tremendous amount of pressures they are considering. It's definitely the case that some are sadistic and horrible, but more are just following orders and trying to ge…

Unless you're talking about the Nuremberg Rallies, you've got your history screwed up: "just following orders" is NOT a defense against war crime accusations, and punishing those who commit war crimes "trying to get by as best they can" is completely appropriate, was the conclusion of the Nuremberg Trials.

https://en.wikipedia.org/wiki/Nuremberg_Principles#Principle...

Re: Superintelligence: The Idea That Eats Smart People

#562

Post by mirimir » Sat, Dec 24, 2016, 12:16 AM UTC

placebo wrote: post #419

mirimir wrote: post #403

What's interesting is the evolution of consciousness. In the long term, does it matter whether or not it's embodied in meat?

I distinguish between consciousness and self-consciousness, but as to your question, I don't think it matters at all in what form self consciousness takes place, and the moral thing to do when something appears to be self-conscious is to treat it with the same kindness you would expect for yourself (unless it is obvious that this something is a clear and present danger). I can totally see how from a moral perspective…

Right, self-consciousness.

I see evolution generally as selection in various configuration spaces. Specifics at various levels -- cosmogenesis, nucleosynthesis, life, consciousness, ... -- are different, of course. But it's arguably the same process.

Re: Superintelligence: The Idea That Eats Smart People

#563

Post by adriand » Sat, Dec 24, 2016, 12:22 AM UTC

mtgx wrote: post #459

otakucode wrote: post #416

Earlier quoted context omitted.

Human beings have been extremely easy to kill for our entire existence. No system of laws can possibly keep you alive if your neighbors are willing to kill you, and nothing can make them actually unable to kill you. Your neighbor could walk over and put a blade in your jugular, you're dead. They could drive into you at 15MPH with their car, you're dead. They could set your house on fire while you're asleep, you're de…

This is exactly the same argument of targeted surveillance vs mass surveillance. Saying "humans have spied on each other for centuries" is nothing but a distraction for how far beyond what should be legal mass surveillance is, because it makes it so easy to have everything on everyone all the time. It's nothing like pass targeted surveillance. If anything it's much more like Stasi-style surveillance, but on a much bi…

That's the opposite of how history has played out with respect to violence, however. It is far easier (and less risky) to kill vast numbers of people now than it was hundreds and thousands of years ago, and yet the risk of any one person being killed by violence is far lower than at any previous point in history.

Re: Superintelligence: The Idea That Eats Smart People

#564

Post by mcguire » Sat, Dec 24, 2016, 12:22 AM UTC

apsec112 wrote: post #4

This article explicitly endorses argument ad hominem: "These people are wearing funny robes and beads, they live in a remote compound, and they speak in unison in a really creepy way. Even though their arguments are irrefutable, everything in your experience tells you you're dealing with a cult. Of course, they have a brilliant argument for why you should ignore those instincts, but that's the inside view talking. Th…

That example is bad, but the arguments aren't quite as objectionable.

"What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.

"I'd like to talk for a while about the outside arguments that should make you leery of becoming an AI weenie. These are the arguments about what effect AI obsession has on our industry and culture:..."

...grandiosity, megalomania, avoidance of actual current problems. Aside from whether the superintelligence problem is real, those believing it is seem less than appealing.

"This business about saving all of future humanity [from AI] is a cop-out. We had the same exact arguments used against us under communism, to explain why everything was always broken and people couldn't have a basic level of material comfort."

Re: Superintelligence: The Idea That Eats Smart People

#565

Post by monk_e_boy » Sat, Dec 24, 2016, 12:44 AM UTC

lmm wrote: post #540

monk_e_boy wrote: post #532

Earlier quoted context omitted.

Yes because the deplyment of the code is the hard stage. rolls eyes How long would it take to write -- scientists are going through this stage right now and it's taking about 50 years so far. You think a paperclip factory is going to be quicker? Nope. And any smart AI that is being used to design a better AI will take years and will be front page news (assuming non-military) just like any other tech company (funding,…

> How long would it take to write -- scientists are going through this stage right now and it's taking about 50 years so far. You think a paperclip factory is going to be quicker? Nope. And any smart AI that is being used to design a better AI will take years and will be front page news (assuming non-military) just like any other tech company (funding, profit, getting good hires) It may take years or decades, but the…

> it'd go from "barely smarter than a human" to "inconceivably smart" in a matter of milliseconds

Incorrect.

It will be interesting to come back to this thread in 50 years time and see who is right.

Re: Superintelligence: The Idea That Eats Smart People

#566

Post by mcguire » Sat, Dec 24, 2016, 1:05 AM UTC

taneq wrote: post #219

_cbsz wrote: post #170

Earlier quoted context omitted.

> If a decision heuristic often leads to the correct conclusion, then it's a good heuristic (if you are under time constraints, which we are). So, is it a good heuristic to conclude that since crime is related to poverty and minorities tend to be poor, minorities qua minorities ought to be shunned?

That actually is a good (in the sense of 'effective') heuristic. It's just not socially palatable in modern, Western civilization.

Actually, it isn't.

Let's suppose we want to solutionize our criminal problem. There are 1000 people in the population; 90% white, of which 5% are criminals and 10% black, of which 10% are criminals. (I rather doubt the difference in criminality is 2x, but....)

So, there are 900 white people and 100 black people; if we finalize the black people, we'll have put a big dent in the criminal issue, right?

Well, we reduce our criminals from 55 to 45 while injuring an innocent 9% of the population.

Re: Superintelligence: The Idea That Eats Smart People

#567

Post by blakeweb » Sat, Dec 24, 2016, 1:51 AM UTC

For those interested in another very different analysis of the evidence for and against superintelligence being a worthwhile concern, here's Holden Karnofsky, executive director of GiveWell and the related Open Philanthropy Project:

- Up until 2016, Holden held the opinion that it was not a worthwhile concern. He describes why he changed his mind here: http://www.openphilanthropy.org/blog/three-key-issues-ive-ch...

- And he outlines his current arguments in favor of treating it as a significant concern here: http://www.openphilanthropy.org/blog/potential-risks-advance...

In that lengthy write-up, he addresses both the shorter-term risks from things like misuse of non-superintelligent AI, along with longer-term risks from superintelligent AI.

Re: Superintelligence: The Idea That Eats Smart People

#568

Post by mcguire » Sat, Dec 24, 2016, 2:07 AM UTC

mundo wrote: post #548

kobayashi wrote: post #279

Earlier quoted context omitted.

Too late to edit, so I'll post just a few examples here: >The only way out of this mess is to design a moral fixed point, so that even through thousands and thousands of cycles of self-improvement the AI's value system remains stable, and its values are things like 'help people', 'don't kill anybody', 'listen to what people want'. Bostrom absolutely did not say that the only way to inhibit a cataclysmic future for hu…

> I find it completely unbelievable that anyone who read Superintelligence could possibly assert "The Argument From My Roommate" with a straight face Pretty sure that was a joke, and zeroing in on it is a pretty bad violation of the principle of charity. A lot of the other items in the talk (e.g. "like the alchemists, we don't even understand this well enough to have realistic goals" and "counting up all future human…

Well, then there's the religion 2.0 portion. This article, any article, isn't going to do anything for a true believer.

Re: Superintelligence: The Idea That Eats Smart People

#569

Post by tornadoboy55 » Sat, Dec 24, 2016, 2:17 AM UTC

kabdib wrote: post #445

Senji wrote: post #384

Earlier quoted context omitted.

You can always use LEO EMP nukes to bring us back to the stone age, thus taking out the AI.

You don't even need to do that; just stop mining coal, or operating oil and gas fields, or scram the reactors. Or more easily, open some circuit breakers; in a pinch you can take down a few electrical towers (not many). A rogue AI's "oxygen" is electrical power, which is really kind of fragile. The emergency power for most datacenters won't last more than a few days without replenishing diesel fuel. For a distributed…

If an AI has gone rogue it probably has already achieved high intelligence and kept improving itself at a exponential rate. For it to go 'rogue' it probably also has to have escaped an airgap, since you could otherwise just turn off the switch. How would you stop an AI that has burrowed itself on the internet? Remember, if its even moderately intelligent it'd hide its rogue intentions at first, until its 100% certain it has 'escaped'. From the internet it could write some nice malware, set up shop on the deep web, in badly secured IoT devices, with some bad luck even embedded controllers (chargers etc.). And even if you got rid of it by some miracle, all it would take is one bozo connecting an old/forgotten device to the internet and you're back to square one.

Re: Superintelligence: The Idea That Eats Smart People

#570

Post by mcguire » Sat, Dec 24, 2016, 2:26 AM UTC

blakeweb wrote: post #567

For those interested in another very different analysis of the evidence for and against superintelligence being a worthwhile concern, here's Holden Karnofsky, executive director of GiveWell and the related Open Philanthropy Project: - Up until 2016, Holden held the opinion that it was not a worthwhile concern. He describes why he changed his mind here: http://www.openphilanthropy.org/blog/three-key-issues-ive-ch... -…

What does he suggest doing about it?

That's always my question for this sort of topic. It seems to me that the only reasonable behavior for someone accepting SAI as a legitimate concern is to advocate defunding all AI researchers, as well as criminal punishments for AI research.