Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apocalypse soon: the scientists preparing for the end times

newstatesman.com

Post reply on HN
Print view View original

21–30 of 62 posts

Re: Apocalypse soon: the scientists preparing for the end times

#21

Post by TeMPOraL »

dosh wrote: post #3

It'd be interesting to learn the depth of study related to spectrum of scenarios in which human race will be facing apocalypse. Few things that pop up in my head: - disease - nuclear (or comparable) war - depletion of natural resources - natural disaster - extra terrestrial influence: asteroid or alien invasion - artificial intelligence - anarchy / disruption of social security hmm...anything else?

Here you go: http://en.wikipedia.org/wiki/Global_catastrophic_risks :).

The keywords are: existential risks.

Re: Apocalypse soon: the scientists preparing for the end times

#22

Post by [deleted] »

chroma wrote: post #2

Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. — Nick Bostrom, Superintelligence: Paths, Dangers, Strategies This is the key point Bostrom is trying…

[deleted]

Re: Apocalypse soon: the scientists preparing for the end times

#23

Post by mariusz79 »

mariusz79 wrote: post #14
TeMPOraL wrote: post #11

Earlier quoted context omitted.

> If you haven't programmed an AI to place sufficient importance on human life (...) And of course you need to watch out lest you overshoot with it and end up like folks in a certain widely popular computer game, where an AI decided that the best way to preserve the diversity of life is to reboot the galaxy every 50 000 years or so. The point of those examples being, the only AI that is safe for us is the one with va…

All these scenarios tend to assume that there is only one AI, or that it can think only one way. AI will grow with us. It will be thought by us.. It's highly unlikely that one day someone will run gcc on ai.cpp and few minutes later the AI will spring to life and start taking over the world. It's more likely that our software will be progressively getting smarter, and our databases larger. We will learn to utilize th…

> after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later.

That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows too fast it will starve the host of these resources which in turn will slow down it's progress. Even if the AI starts on a supercomputer in some university somewhere and decides that it needs more power, most likely it will start by "infecting" other computers on the network. These computers will not be nearly powerful enough to add much to the current processing power of the supercomputer. Such invasion is likely to get noticed. Each infected node will not be another super intelligent being, but due to hardware limitations it will most likely be a software that's limited in scope and designed to propagate and control the resources, linking them with the main host. We can safely assume that restarting, or reformatting the drives will restore the node to it's non-infected state thus slowing down the "invasion". Why we can assume that? It will take AI long time to learn everything about every architecture and operating system and apply that knowledge. The AI that's hosted on a supercomputer will not be able to suddenly start rewriting its software, take over nearby nodes, learn everything about our world etc.

>You're assuming that AI will grow with us and necessarily reach the same value system we have (one we don't really comprehend ourselves, mind you).

No I'm not. Only thing I can assume is that if the AI is smart enough to figure out how "convert" humans to paper clips, it will be smart enough to understand that it's pointless. It may decide to use it's super intelligence to take over the world, and I can't argue with that.. But if it continues to make paper clips even if they are no longer necessary then by definition this is not AI, just a stupid replicator.

Re: Apocalypse soon: the scientists preparing for the end times

#24

Post by Aardwolf »

dosh wrote: post #3

It'd be interesting to learn the depth of study related to spectrum of scenarios in which human race will be facing apocalypse. Few things that pop up in my head: - disease - nuclear (or comparable) war - depletion of natural resources - natural disaster - extra terrestrial influence: asteroid or alien invasion - artificial intelligence - anarchy / disruption of social security hmm...anything else?

Out of all of those, artificual intelligence is the only one where at least something continues to exist.

Re: Apocalypse soon: the scientists preparing for the end times

#25

Post by TeMPOraL »

mariusz79 wrote: post #23
mariusz79 wrote: post #14

Earlier quoted context omitted.

All these scenarios tend to assume that there is only one AI, or that it can think only one way. AI will grow with us. It will be thought by us.. It's highly unlikely that one day someone will run gcc on ai.cpp and few minutes later the AI will spring to life and start taking over the world. It's more likely that our software will be progressively getting smarter, and our databases larger. We will learn to utilize th…

> after that a wrongly programmed AI may just as well "spring to life and start taking over the world" few minutes later. That's physically impossible. AI still needs to improve incrementally and this will take time and resources. We can consider such AI to act like a virus - it will need a host to grow - in this case computer resources. It will be limited by the CPU, network speeds, storage and power. If it grows to…

You're assuming quite a lot about how such event would develop. Keep in mind that a sufficiently smart AI wouldn't have to infect stuff, it could just ask. Think about IQ 200 sociopath that has Internet plugged into its head. It would be extremely difficult to outsmart it.

> it will be smart enough to understand that it's pointless.

Pointless for humans, not for an AI. You're projecting a lot of human values and thinking on AI. See my comment [0].

[0] - https://news.ycombinator.com/item?id=8387771

Re: Apocalypse soon: the scientists preparing for the end times

#26

Post by Zigurd »

dosh wrote: post #3

It'd be interesting to learn the depth of study related to spectrum of scenarios in which human race will be facing apocalypse. Few things that pop up in my head: - disease - nuclear (or comparable) war - depletion of natural resources - natural disaster - extra terrestrial influence: asteroid or alien invasion - artificial intelligence - anarchy / disruption of social security hmm...anything else?

Is anarchy more dangerous than corporatism?

Re: Apocalypse soon: the scientists preparing for the end times

#27

Post by hawleyal »

mariusz79 wrote: post #19
DanAndersen wrote: post #13

Earlier quoted context omitted.

One of the main points of Bostrom's work is the orthogonality thesis, which is that intelligence is orthogonal to any particular set of values -- basically, all those nice fuzzy things like caring what the humans actually meant? You don't get those for free when you build intelligence. It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little h…

>Think about modern financial trading algorithm They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks. Paper Clip machine will be a robot, not AI.

The point still stands.

I have written algorithms for autonomous programs like this. Pretty much advanced IFTTT, automatically analyzing the situation and reacting.

The program is not aware of any values other than those it is programmed. Obviously, it will not regard or reflect upon its actions and the effects thereof outside of optimizing the profit from trading.

Re: Apocalypse soon: the scientists preparing for the end times

#28

Post by mariusz79 »

mariusz79 wrote: post #19
DanAndersen wrote: post #13

Earlier quoted context omitted.

One of the main points of Bostrom's work is the orthogonality thesis, which is that intelligence is orthogonal to any particular set of values -- basically, all those nice fuzzy things like caring what the humans actually meant? You don't get those for free when you build intelligence. It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little h…

>Think about modern financial trading algorithm They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks. Paper Clip machine will be a robot, not AI.

@hawleyal yes, and I understand that. I'm working in industrial automation, but Bostrom talks about "superintelligent paper-clip maker which works out that it could create more paper clips by extracting carbon atoms from human bodies". Your algorithm is not super intelligent is it? It's just a dumb program executing operations according to some predefined constraints. It may brake the financial system if you don't set any rules to prevent that, but it will not suddenly decide to start trading in gold if you designed it to trade on forex. If it suddenly decided that Forex is a better option to make money, then we could consider it intelligent and start worrying about it. We would really have to worry if it decided that the best way to become the richest person on the planet is to kill everyone on it. But that would require super intelligence.

My problem with this whole example given by Bostrom is that if the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless.

I fully understand that this paper clip example is just oversimplification, but I would expect more from people spending entire lives thinking about that stuff.

It's nothing but a fear-porn designed to get more funding.

Re: Apocalypse soon: the scientists preparing for the end times

#29

Post by Zigurd »

chroma wrote: post #2

Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. — Nick Bostrom, Superintelligence: Paths, Dangers, Strategies This is the key point Bostrom is trying…

> Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions.

He gets halfway to right. Recently there was a thread about suicide on HN. Nearly all the comments on that thread ignored that humans, like other animals, are built to propagate the information in their genes, and argued against suicide based on concepts that are driven by or that rationalize subconscious drives. An AI, even one with a programmed goal of propagating itself, does not carry information in genes and the information in an AI was never selected for the actual ability to propagate in a larger system that thinks it is in control of its goals.

Real AIs are likely to be extremely alien. Even in warning people about anthropomorpizing, there's a lot of anthropomorphism. Evil AIs are depicted as human evil. So much of human motivation is closed to introspection that imagining alien and evil motivation is unlikely to be close to the eventual reality. This probably won't be widely understood until AIs are built. And what if they all choose to erase themsleves?

Re: Apocalypse soon: the scientists preparing for the end times

#30

Post by fauigerzigerk »

DanAndersen wrote: post #13
mariusz79 wrote: post #8

Earlier quoted context omitted.

I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-clips does not require super intelligence, just a simple automaton. If we however do employ AI to make paper-clip it should know how many we need on average, and what are they used for. If the paper clips are for our (human) use, the AI will be really dumb if…

One of the main points of Bostrom's work is the orthogonality thesis, which is that intelligence is orthogonal to any particular set of values -- basically, all those nice fuzzy things like caring what the humans actually meant? You don't get those for free when you build intelligence. It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little h…

I agree that an AGI wouldn't necessarily have to share our values. But what Bostrom is describing - an intelligence so narrowly focused on making paper clips that it destroys humanity as a side-effect - does not deserve the name AGI.

Any definition of "general intelligence" would have to include the ability to revise and reconsider ones own goals and purpose. An AGI that is forever bound to maximize the paper clip output at any cost, does not, in my view, meet that definition.

But I grant you that there could be "higher" goals that are incompatible with the existence of humanity. Making paper clips just is not it.

Post reply on HN