Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apocalypse soon: the scientists preparing for the end times

newstatesman.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: Apocalypse soon: the scientists preparing for the end times

#51

Post by JulianMorrison »

api wrote: post #50
JulianMorrison wrote: post #48

Earlier quoted context omitted.

Let me put it this way. Why do humans continue to do all sorts of obviously stupid counterproductive things, at large and small scales? We know that reproducing so many of us is stupid. We know that eating so much fat, sugar and salt is stupid. We know that overheating the atmosphere is stupid. We know that destroying biodiversity, overfishing to the point of extinction, trashing the forests we need to breathe etc et…

I guess my assumption is that increased intelligence should bring with it increased self-awareness. If it doesn't, then... well... like I said elsewhere in the thread... put that on your list of potential depressing answers to the Fermi paradox.

Your mistake is assuming that self awareness is sufficient. "To know" is separated from "to choose to act" by preference, and preference is based on goals, and the ultimate, root level goals of a being are not likely to be something it wants to rewrite, even if it has the ability, any more than you'd willingly take a pill to become inhuman.

Re: Apocalypse soon: the scientists preparing for the end times

#52

Post by mariusz79 »

JulianMorrison wrote: post #48
mariusz79 wrote: post #43

Earlier quoted context omitted.

I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources. Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us al…

Let me put it this way. Why do humans continue to do all sorts of obviously stupid counterproductive things, at large and small scales? We know that reproducing so many of us is stupid. We know that eating so much fat, sugar and salt is stupid. We know that overheating the atmosphere is stupid. We know that destroying biodiversity, overfishing to the point of extinction, trashing the forests we need to breathe etc et…

But we are not "super intelligent" ;) The truth is that this whole discussion does not make much sense if we have different understanding of AI. For me a super intelligent AI is an entity that is able to reason, notice any potential problems and come up with solutions. Paper Clip AI on the other hand is smart enough to know how to create paper clips and create whole manufacturing process of converting humans into paper clips but stupid enough not to understand, or care why it's doing that and not knowing that these paper clips will have no use if there are no humans left.

In my opinion one can't talk about a super intelligent being and not allow that being to be able to reason and understand it's place in the world.

Re: Apocalypse soon: the scientists preparing for the end times

#53

Post by mariusz79 »

JamesArgo wrote: post #47
mariusz79 wrote: post #43

Earlier quoted context omitted.

I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources. Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us al…

You've got this normative halo around "intelligence". All intelligence is, is an agent's ability to satisfy its utility function. >But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent? That is the correct action if all it cares about is making paperclips. Creating paperclips satisfies its utility function. Fretting over the ex…

Is Ebola virus intelligent? It has a goal - reproduce - and it's trying to maximize it through any means available to it.I don't think we can say that the virus is intelligent. If it continues to evolve, at some point in time it could potential (highly unlikely but still possible) become intelligent, but will it become self-aware at this point? And if it does will it still want to replicate uncontrollably or will, just like humans, decide the it needs to better manage it's environment?

Re: Apocalypse soon: the scientists preparing for the end times

#54

Post by Vektorweg »

dosh wrote: post #3

It'd be interesting to learn the depth of study related to spectrum of scenarios in which human race will be facing apocalypse. Few things that pop up in my head: - disease - nuclear (or comparable) war - depletion of natural resources - natural disaster - extra terrestrial influence: asteroid or alien invasion - artificial intelligence - anarchy / disruption of social security hmm...anything else?

A critical existence failure.

Re: Apocalypse soon: the scientists preparing for the end times

#55

Post by mapt »

fauigerzigerk wrote: post #30
DanAndersen wrote: post #13

Earlier quoted context omitted.

One of the main points of Bostrom's work is the orthogonality thesis, which is that intelligence is orthogonal to any particular set of values -- basically, all those nice fuzzy things like caring what the humans actually meant? You don't get those for free when you build intelligence. It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little h…

I agree that an AGI wouldn't necessarily have to share our values. But what Bostrom is describing - an intelligence so narrowly focused on making paper clips that it destroys humanity as a side-effect - does not deserve the name AGI. Any definition of "general intelligence" would have to include the ability to revise and reconsider ones own goals and purpose. An AGI that is forever bound to maximize the paper clip ou…

It absolutely does. The point of it all is that setting goals we would be happy with requires an inordinately stronger theory of humanity. ANY goal we give an AGI that is smarter than ourselves and highly scalable will be pursued with ruthless efficiency in ways we can't imagine. The ways we can imagine this sort of thing going wrong abound in science fiction. A paperclip optimizer is a thought experiment about an AGI deployed to a simple, trivial goal without due consideration of how to goal-set.

There is a whole world of more complex failure modes. "Make everyone happy" leads to wireheading or heroin addiction for the population, unless you grant a more compelling Theory of Happiness than any two human beings can agree on. "Make everyone food to eat", "Cure my PTSD", "Set up a colony on Mars" - unless you define all these things in a way that is specific not just to humans, but to modern culture, you leave open the possibility of undesired results. Mythology is thick with gods and djinn who grant wishes but turn out to be tricksters.

It's not that it "wouldn't necessarily have to share our values" - it's that there's no way in hell these things would share our values; They actually can't share our values until we solve "our values" as a complete mathematical proof, and then figure out a way to code that. In a resilient way, without any chance of the neuroses or obsessions or self-deceptions or delusions or corruptions of perspective available to humans.

Solving philosophy is a precursor to a Friendly AGI.

Good luck bug-fixing.

Re: Apocalypse soon: the scientists preparing for the end times

#56

Post by fauigerzigerk »

mapt wrote: post #55
fauigerzigerk wrote: post #30

Earlier quoted context omitted.

I agree that an AGI wouldn't necessarily have to share our values. But what Bostrom is describing - an intelligence so narrowly focused on making paper clips that it destroys humanity as a side-effect - does not deserve the name AGI. Any definition of "general intelligence" would have to include the ability to revise and reconsider ones own goals and purpose. An AGI that is forever bound to maximize the paper clip ou…

It absolutely does. The point of it all is that setting goals we would be happy with requires an inordinately stronger theory of humanity. ANY goal we give an AGI that is smarter than ourselves and highly scalable will be pursued with ruthless efficiency in ways we can't imagine. The ways we can imagine this sort of thing going wrong abound in science fiction. A paperclip optimizer is a thought experiment about an AG…

I totally agree that it could go wrong in many ways. We could end up in a zoo or go extinct, no doubt about that. But it seems to me that there is a bit of a mechanistic undertone in phrases like "ruthless efficiency" as well as in the paper clip example and that contradicts what we consider intelligent.

General artificial intelligence is an antropomorphic concept in the first place. It includes inefficiencies. It includes self reflection. It includes uncertainty and disagreement about what is and isn't efficient and about what is and isn't right.

Does that guarantee that any AGI would be reasonable or even morally acceptable in our sense? No. But if it is intelligent in our sense we know two things:

This entity can be reasoned with, and having grown up in our midst, it will understand us. It would also be very surprising if it shared none of our values. After all it has evolved among us and any intelligent being is going to pick up some of the values of the society it is part of.

Secondly, intelligence includes creative leaps. That means, no two AGIs would share the exact same opinion, just as no two humans do. So we may be able to convince a majority of AGIs to be kind to us, and any efficiency of an AGI society wouldn't be as ruthlessly efficient as if there were just one single all powerful AGI.

So, all I'm saying is, let's not strip the term general intelligence of all its non-mechanistic, self-referential, social traits. An AGI could one day anihilate us but it would probably not be a side-effect of some fairly unimportant industrial process like making paper clips.

Re: Apocalypse soon: the scientists preparing for the end times

#57

Post by sitkack »

wavegeek wrote: post #31
mariusz79 wrote: post #8

Earlier quoted context omitted.

I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-clips does not require super intelligence, just a simple automaton. If we however do employ AI to make paper-clip it should know how many we need on average, and what are they used for. If the paper clips are for our (human) use, the AI will be really dumb if…

For goodness sake the paperclip maker is an example . The point is that unless you specify the goals very carefully, you may get what you asked for, not what you wanted.

By analogy the AI is a leprechaun or a genie that will give you what you asked for but in a way that could do you harm.

Re: Apocalypse soon: the scientists preparing for the end times

#58

Post by Houshalter »

api wrote: post #38
chroma wrote: post #2

Far too often, people anthropomorphize AI or draw from fiction. Bostrom does his best to dispel these notions. The algorithm just does what it does; and unless it is a very special kind of algorithm, it does not care that we clasp our heads and gasp in dumbstruck horror at the absurd inappropriateness of its actions. — Nick Bostrom, Superintelligence: Paths, Dangers, Strategies This is the key point Bostrom is trying…

I think paperclip optimizers and such conflict with the definition of intelligence and sentience. A truly intelligent, sentient paperclip optimizer would begin to ponder existential questions like "why am I making paperclips? what's so important about paperclips?"

There is a distinction between "terminal" and "instrumental" values. If you continually ask yourself questions like "why am I doing this", you will eventually get an answer of "just because". That is your terminal value, the reason you do everything you do.

Humans have some complicated unknown set of terminal values. Everything we do, we do because we believe it will maximize our values. A value might be something like "avoid suffering", or "other people suffering is bad", etc.

In this example the AIs terminal value is "make paperclips". Everything it does, it does because it believes it will maximize this value. It would never ask itself questions like "why", because that action doesn't maximize that value.

If it has the ability to rewrite it's own code, it would rewrite it in such a way that it can't ever change it's own goals. Changing goals is counterproductive to achieving them.

Re: Apocalypse soon: the scientists preparing for the end times

#59

Post by Houshalter »

Aardwolf wrote: post #24
dosh wrote: post #3

It'd be interesting to learn the depth of study related to spectrum of scenarios in which human race will be facing apocalypse. Few things that pop up in my head: - disease - nuclear (or comparable) war - depletion of natural resources - natural disaster - extra terrestrial influence: asteroid or alien invasion - artificial intelligence - anarchy / disruption of social security hmm...anything else?

Out of all of those, artificual intelligence is the only one where at least something continues to exist.

I think you mean the other way around. All the ones listed will very likely have at least some human survivors. Eventually humanity will rebuild and get back to where it was more or less.

AI is the only thing listed with the capability to literally destroy the planet. While something might still exist afterwards, it will likely be totally meaningless and worthless: http://lesswrong.com/lw/y3/value_is_fragile/

Re: Apocalypse soon: the scientists preparing for the end times

#60

Post by kordless »

TeMPOraL wrote: post #34
kordless wrote: post #33

Earlier quoted context omitted.

His assumptions are logically sound given our universe's rules. There is no such thing as a free lunch here. The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exi…

You're being sarcastic, right? Since when our universe is governed by capitalist economy?

I'm quite serious. Our universe is governed by laws of conservation and efficiencies. For AI to exist, it needs power. Power must come from somewhere in the universe and, at least here on earth, it has been shown that there is competition for those resources. Whether this alludes to business or not is irrelevant; it's a fact of our reality that we are all competing against each other for resources.
Post reply on HN