Apocalypse soon: the scientists preparing for the end times • Hacker News Board

Re: Apocalypse soon: the scientists preparing for the end times

#31

Post by wavegeek » Tue, Sep 30, 2014, 1:33 PM UTC

mariusz79 wrote: post #8

chroma wrote: post #2

Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. — Nick Bostrom, Superintelligence: Paths, Dangers, Strategies This is the key point Bostrom is trying…

I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-clips does not require super intelligence, just a simple automaton. If we however do employ AI to make paper-clip it should know how many we need on average, and what are they used for. If the paper clips are for our (human) use, the AI will be really dumb if…

For goodness sake the paperclip maker is an example. The point is that unless you specify the goals very carefully, you may get what you asked for, not what you wanted.

Re: Apocalypse soon: the scientists preparing for the end times

#32

Post by SuperChihuahua » Tue, Sep 30, 2014, 1:33 PM UTC

dosh wrote: post #3

It'd be interesting to learn the depth of study related to spectrum of scenarios in which human race will be facing apocalypse. Few things that pop up in my head: - disease - nuclear (or comparable) war - depletion of natural resources - natural disaster - extra terrestrial influence: asteroid or alien invasion - artificial intelligence - anarchy / disruption of social security hmm...anything else?

To quote a guy called Elon Musk:

"Things have happened quickly. It took us millions of years to evolve into what we are, but in the last sixty years, with atomic weaponry, we've created the potential to extinguish ourselves. And if it's not us, it will inevitably be something else. If not a meteorite in the relative short term, then the expansion of the Sun's corona. It will happen."

He has also said that there are a number of other threats with the ability to extinct life on Earth. They include an engineered virus unleashed by terrorists, an inadvertent creation of a micro black hole, or a yet unknown technology.

Re: Apocalypse soon: the scientists preparing for the end times

#33

Post by kordless » Tue, Sep 30, 2014, 1:36 PM UTC

TeMPOraL wrote: post #25

mariusz79 wrote: post #23

Earlier quoted context omitted.

> after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later. That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows to…

You're assuming quite a lot about how such event would develop. Keep in mind that a sufficiently smart AI wouldn't have to infect stuff, it could just ask . Think about IQ 200 sociopath that has Internet plugged into its head. It would be extremely difficult to outsmart it. > it will be smart enough to understand that it's pointless. Pointless for humans, not for an AI. You're projecting a lot of human values and thi…

His assumptions are logically sound given our universe's rules. There is no such thing as a free lunch here. The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exist here. I'm sure as shit not going to buy a paper clip made out of you!

Re: Apocalypse soon: the scientists preparing for the end times

#34

Post by TeMPOraL » Tue, Sep 30, 2014, 1:45 PM UTC

kordless wrote: post #33

TeMPOraL wrote: post #25

Earlier quoted context omitted.

You're assuming quite a lot about how such event would develop. Keep in mind that a sufficiently smart AI wouldn't have to infect stuff, it could just ask . Think about IQ 200 sociopath that has Internet plugged into its head. It would be extremely difficult to outsmart it. > it will be smart enough to understand that it's pointless. Pointless for humans, not for an AI. You're projecting a lot of human values and thi…

His assumptions are logically sound given our universe's rules. There is no such thing as a free lunch here. The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exi…

You're being sarcastic, right? Since when our universe is governed by capitalist economy?

Re: Apocalypse soon: the scientists preparing for the end times

#35

Post by mariusz79 » Tue, Sep 30, 2014, 1:49 PM UTC

TeMPOraL wrote: post #25

mariusz79 wrote: post #23

Earlier quoted context omitted.

> after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later. That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows to…

You're assuming quite a lot about how such event would develop. Keep in mind that a sufficiently smart AI wouldn't have to infect stuff, it could just ask . Think about IQ 200 sociopath that has Internet plugged into its head. It would be extremely difficult to outsmart it. > it will be smart enough to understand that it's pointless. Pointless for humans, not for an AI. You're projecting a lot of human values and thi…

We're designing the AI so it's safe to assume that the first versions of it will think like a human. We don't know how it will think after it starts to evolve but for the Paper Clip AI scenario that' irrelevant.

The AI, even the Paper Clips AI , will need some basic understanding of our world to make the clips. It needs to understand what paper clips are. It needs to understand what they are made from. It needs to understand the process of making paper clips. where to get the material for the paper clips. how to pay for that material. It has to think like us, or at least understand our thought process. It will need to know how many we need, what colors, sizes. Where to ship them.

We can't think of AI as a god. It won't suddenly know everything and understand what we want without our input. It will be limited by the resources given to it.

Even if we tell the Paper Clip AI - "Go make me some paper clips" do you really expect it to start calling banks to get the loan for the material, arrange shipment and unloading it. Will it also order the machines, call people to have them connected, hire workers to do it? If it does all of that will it also take care of sales? Marketing? Will it understand the supply and demand? To do all of this it needs to understand us, and our world. If at this point it decides that it makes sense to convert us to paper clips, then perhaps it's right and we deserve this.

Re: Apocalypse soon: the scientists preparing for the end times

#36

Post by MoistDinosaur » Tue, Sep 30, 2014, 1:54 PM UTC

Havvy wrote: post #5

dosh wrote: post #3

It'd be interesting to learn the depth of study related to spectrum of scenarios in which human race will be facing apocalypse. Few things that pop up in my head: - disease - nuclear (or comparable) war - depletion of natural resources - natural disaster - extra terrestrial influence: asteroid or alien invasion - artificial intelligence - anarchy / disruption of social security hmm...anything else?

Anarchy won't cause an apocalypse, just a change in the structure of the ruling class as a new ruling structure is created. also on the extra terrestrial influence side, there's direct hits by gamma rays that could wipe out Earth in an instant.

Anarchy causing a collapse of the ruling structure could create a war that destroys much of society. Even without anarchy, imagine a country like China or the US plunged into a civil war. The effects on the global economy would be disastrous

Re: Apocalypse soon: the scientists preparing for the end times

#37

Post by mariusz79 » Tue, Sep 30, 2014, 1:56 PM UTC

wavegeek wrote: post #31

mariusz79 wrote: post #8

Earlier quoted context omitted.

I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-clips does not require super intelligence, just a simple automaton. If we however do employ AI to make paper-clip it should know how many we need on average, and what are they used for. If the paper clips are for our (human) use, the AI will be really dumb if…

For goodness sake the paperclip maker is an example . The point is that unless you specify the goals very carefully, you may get what you asked for, not what you wanted.

no shi* that it's an example, but it's a stupid one. If you have to specify the goals very carefully then the AI it's not super intelligent by definition. It means that not only it does not understand what it's asked for, but does not understand our world. I can assume that none of us here can be qualified as super intelligent, yet we know what to do when asked to start producing paper clips, or at least we know where to start.

Re: Apocalypse soon: the scientists preparing for the end times

#38

Post by api » Tue, Sep 30, 2014, 1:57 PM UTC

chroma wrote: post #2

Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. — Nick Bostrom, Superintelligence: Paths, Dangers, Strategies This is the key point Bostrom is trying…

I think paperclip optimizers and such conflict with the definition of intelligence and sentience. A truly intelligent, sentient paperclip optimizer would begin to ponder existential questions like "why am I making paperclips? what's so important about paperclips?"

Re: Apocalypse soon: the scientists preparing for the end times

#39

Post by JulianMorrison » Tue, Sep 30, 2014, 2:07 PM UTC

api wrote: post #38

chroma wrote: post #2

Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. — Nick Bostrom, Superintelligence: Paths, Dangers, Strategies This is the key point Bostrom is trying…

I think paperclip optimizers and such conflict with the definition of intelligence and sentience. A truly intelligent, sentient paperclip optimizer would begin to ponder existential questions like "why am I making paperclips? what's so important about paperclips?"

You fail to understand that humans come pre-loaded with sociability and goodness goals. Those few who don't, we rightly regard as varelse.

A sufficiently smart rogue general AI will indeed understand that it is doing wrong, harmful, senseless things. It just won't care.

Re: Apocalypse soon: the scientists preparing for the end times

#40

Post by veddan » Tue, Sep 30, 2014, 2:07 PM UTC

api wrote: post #38

chroma wrote: post #2

Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. — Nick Bostrom, Superintelligence: Paths, Dangers, Strategies This is the key point Bostrom is trying…

I think paperclip optimizers and such conflict with the definition of intelligence and sentience. A truly intelligent, sentient paperclip optimizer would begin to ponder existential questions like "why am I making paperclips? what's so important about paperclips?"

I'm sure a similar argument could be made about many human activities.