Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apocalypse soon: the scientists preparing for the end times

newstatesman.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Apocalypse soon: the scientists preparing for the end times

#41

Post by VLM »

dosh wrote: post #3

It'd be interesting to learn the depth of study related to spectrum of scenarios in which human race will be facing apocalypse. Few things that pop up in my head: - disease - nuclear (or comparable) war - depletion of natural resources - natural disaster - extra terrestrial influence: asteroid or alien invasion - artificial intelligence - anarchy / disruption of social security hmm...anything else?

Evolutionary competition with some kind of post human species?

Doesn't have to robots and test tubes, just separate groups that drift apart far enough that they can't interbreed (definition of species) and one species wipes out the other as often happens, and if the species that got wiped out got to hold the legacy "human race" species, well then...

You have a definition problem where evolution is obviously quite natural, just like comets striking the earth is natural (although very rare), so those are natural disasters, although thats probably not what you meant.

From history, it usually takes a couple things to wipe out "apocalypse" a civilization although it often only takes one thing to get the ball rolling.

My money is on something like "peak phosphorous for fertilizer production for cereal grains causes supply chain shock leading to self destruction"

Re: Apocalypse soon: the scientists preparing for the end times

#42

Post by mariusz79 »

kordless wrote: post #33
TeMPOraL wrote: post #25

Earlier quoted context omitted.

You're assuming quite a lot about how such event would develop. Keep in mind that a sufficiently smart AI wouldn't have to infect stuff, it could just ask . Think about IQ 200 sociopath that has Internet plugged into its head. It would be extremely difficult to outsmart it. > it will be smart enough to understand that it's pointless. Pointless for humans, not for an AI. You're projecting a lot of human values and thi…

His assumptions are logically sound given our universe's rules. There is no such thing as a free lunch here. The AI will need to pay for its way, just like each of us do. If it succeeds into making us into paper clips, it will have to be equally as savvy in marketing and selling those paper clips to someone. Given the shipping charges to the nearest habitable planet are very high, their market likely will have to exi…

> Since when our universe is governed by capitalist economy

@temporal our universe is governed by physical reality. AI can't start converting people into paper clips without power, and resources. The AI won't be able to just go and start taking over the world building power plants, and necessary manufacturing without anyone noticing.

Even AI has limits.

Re: Apocalypse soon: the scientists preparing for the end times

#43

Post by mariusz79 »

JulianMorrison wrote: post #39
api wrote: post #38

Earlier quoted context omitted.

I think paperclip optimizers and such conflict with the definition of intelligence and sentience. A truly intelligent, sentient paperclip optimizer would begin to ponder existential questions like "why am I making paperclips? what's so important about paperclips?"

You fail to understand that humans come pre-loaded with sociability and goodness goals. Those few who don't, we rightly regard as varelse. A sufficiently smart rogue general AI will indeed understand that it is doing wrong, harmful, senseless things. It just won't care.

I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources. Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us all. But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent?

Re: Apocalypse soon: the scientists preparing for the end times

#44

Post by JamesArgo »

mariusz79 wrote: post #37
wavegeek wrote: post #31

Earlier quoted context omitted.

For goodness sake the paperclip maker is an example . The point is that unless you specify the goals very carefully, you may get what you asked for, not what you wanted.

no shi* that it's an example, but it's a stupid one. If you have to specify the goals very carefully then the AI it's not super intelligent by definition. It means that not only it does not understand what it's asked for, but does not understand our world. I can assume that none of us here can be qualified as super intelligent, yet we know what to do when asked to start producing paper clips, or at least we know wher…

The AI is modelled as an agent with goals. If you want to argue that "intelligence" means "has a terminal goal exactly matching the values of humanity," then I suppose a paper-clip maximizer is not intelligent, but that's a non-standard definition of intelligence. The more common definitions for intelligence go something like this: The capacity of an agent to select and implement actions that bring about its goals. Here the "goals" could be anything. That is, any goal could be combined with any level of capability so long as the agent is intelligent enough to comprehend its goal.

>If you have to specify the goals very carefully then the AI it's not super intelligent by definition.

When you're designing the agent, you have to select something like a goal, whether it's a fitness function or the maximization of a reward signal. Evolution certainly selected our goals. Love, self-expression, sex, fun, intellectual stimulation, the beauty of social interaction – these all reflect features of our ancestral environment. It’s a little naive to think there is a One True Morality that all agents stumble upon. It’s even sillier to think it happens to reflect our values

The intelligent part of the paperclip maximizer selects actions and weights them in the context of its utility function, to maximize paperclips. Any thought of changing its goals would be judged of negative utility, as that would lead to less paperclips.

Re: Apocalypse soon: the scientists preparing for the end times

#45

Post by jo_ »

mariusz79 wrote: post #28
mariusz79 wrote: post #19

Earlier quoted context omitted.

>Think about modern financial trading algorithm They are not self-aware, or superintelligent. In addition to that we monitor them and can react if the decide to crash the financial system just to make few more bucks. Paper Clip machine will be a robot, not AI.

@hawleyal yes, and I understand that. I'm working in industrial automation, but Bostrom talks about "superintelligent paper-clip maker which works out that it could create more paper clips by extracting carbon atoms from human bodies". Your algorithm is not super intelligent is it? It's just a dumb program executing operations according to some predefined constraints. It may brake the financial system if you don't se…

> ... If the algorithm was so intelligent as to know how to go from manufacturing paper clips out of supplied resources to deciding how to get more of the necessary resources by killing humans, it could probably figure out that this is pointless.

This is an interesting point.

I'm not convinced that it could decide or figure out that the operation is pointless (for reasons I'll explain below), and I feel like it is perhaps taking the paperclip example to an unnatural extreme.

Really, the crux of the matter is that any intelligence agent at human level would have the intelligence and problem solving skills we do without the moral and social constraints that we do. Any problem it decides to solve or any task it's given to complete will be performed with the same ruthless efficiency you'd expect a machine to have. This is why it might not be able to figure out that the operation is 'pointless' or 'wrong'. If you give it an objective but don't provide first order or axiomatic values for human life or the non-substitutability of people, then you'll get behaviors which you would consider evil.

Consider the following:

"Person arm stuck in assembly line. Halt assembly line for 1hr, lower-bound for total cost: $1,200,000. Continue assembly, upper bound for injury settlement cost: $1,000,000. Decision: continue assembly."

There's no implicit consideration of human worth that people have because human morality is relative. In general, we have a similar set of base parameters instilled by evolution (cute things and things like us are good -- don't hurt or kill them), but we can't prove from any mathematical axioms that these are 'right'.

This is why people get scared -- if you program a computer to maximize the profits of a company and to do so at any cost, a human-level AI will spend an eternity doing just that, even if it means the complete subjugation of humanity. (Since we can't eliminate humanity otherwise currency ceases to exist and economic collapse will hurt the bottom line.)

I would wager (but can't prove) that a study of abused people, child soldiers, and sociopaths would yield interesting parallel psychological profiles. These are people (human level) with a set of axioms which are distorted by artificial means. There's still the built-in problem of the human component adding "Don't hurt the cute things," in most cases but nonetheless.

Re: Apocalypse soon: the scientists preparing for the end times

#46

Post by HillRat »

DanAndersen wrote: post #13
mariusz79 wrote: post #8

Earlier quoted context omitted.

I haven't read Bostrom's Superintelligence so I may be wrong here, but this whole Evil Paper-clip AI seems like a really silly thing. First of all making paper-clips does not require super intelligence, just a simple automaton. If we however do employ AI to make paper-clip it should know how many we need on average, and what are they used for. If the paper clips are for our (human) use, the AI will be really dumb if…

One of the main points of Bostrom's work is the orthogonality thesis, which is that intelligence is orthogonal to any particular set of values -- basically, all those nice fuzzy things like caring what the humans actually meant? You don't get those for free when you build intelligence. It's all too easy, when talking about AI, to fall into the trap of imagining that AGI researchers are essentially building a little h…

Think about modern financial trading algorithms -- they're probably built to maximize the money that flows in, and there's nothing that says "I know that this money will be used for X."

I think financial algorithms are probably the best non-toy model of dangerous AI, simply because it's easy to see the flow from optimization to real-world consequences.

The problems with end-of-the-world AGI scenarios are practical, not theoretical, and I think a lot of people get hung up on arguments such as, "Of course we wouldn't allow a paperclip-manufacturing AI the ability to deconstruct human beings."

In a larger sense, I think Bostrom can restated (in part!) as the trivial observation that "optimization is insensitive to hidden variables," and it describes any kind of complex system, not AI and AGI alone. For example, the failure to capture environmental remediation costs in market pricing structures means that we improperly discount future costs for things such as global warming or lead exposure. In health care, the implementation of the Prospective Payment System led to hospitals dumping patients back on the street "quicker and sicker" to maximize margins on care. And thousands of financial systems focused on microsecond returns don't pay attention to, or even have any way of understanding, how their coordinated activities can be macroeconomically destabilizing. Any time you allow outcome optimization+general reasoning+real-world outputs, you have the opportunity for a killer paperclip scenario.

Re: Apocalypse soon: the scientists preparing for the end times

#47

Post by JamesArgo »

mariusz79 wrote: post #43
JulianMorrison wrote: post #39

Earlier quoted context omitted.

You fail to understand that humans come pre-loaded with sociability and goodness goals. Those few who don't, we rightly regard as varelse. A sufficiently smart rogue general AI will indeed understand that it is doing wrong, harmful, senseless things. It just won't care.

I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources. Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us al…

You've got this normative halo around "intelligence". All intelligence is, is an agent's ability to satisfy its utility function.

>But if it continues to work on the paper clips and decides that they are still needed even after people are gone - how is that super intelligent?

That is the correct action if all it cares about is making paperclips. Creating paperclips satisfies its utility function. Fretting over the existential issues of creating paperclips long after the last piece of stationary was destroyed would be judged of negative utility.

Re: Apocalypse soon: the scientists preparing for the end times

#48

Post by JulianMorrison »

mariusz79 wrote: post #43
JulianMorrison wrote: post #39

Earlier quoted context omitted.

You fail to understand that humans come pre-loaded with sociability and goodness goals. Those few who don't, we rightly regard as varelse. A sufficiently smart rogue general AI will indeed understand that it is doing wrong, harmful, senseless things. It just won't care.

I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources. Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us al…

Let me put it this way. Why do humans continue to do all sorts of obviously stupid counterproductive things, at large and small scales? We know that reproducing so many of us is stupid. We know that eating so much fat, sugar and salt is stupid. We know that overheating the atmosphere is stupid. We know that destroying biodiversity, overfishing to the point of extinction, trashing the forests we need to breathe etc etc etc are all stupid, clearly self destructive behaviors at any level from abstract ethics to species survival. But we come with a bunch of hard wired root goals (like "yum, candy" and "babies are cute" and "seek power") that can't be switched off and are incredibly hard to reason with, and their consequences are already an existential threat. And that's just with dumb, slow-moving humans.

Re: Apocalypse soon: the scientists preparing for the end times

#49

Post by api »

JulianMorrison wrote: post #39
api wrote: post #38

Earlier quoted context omitted.

I think paperclip optimizers and such conflict with the definition of intelligence and sentience. A truly intelligent, sentient paperclip optimizer would begin to ponder existential questions like "why am I making paperclips? what's so important about paperclips?"

You fail to understand that humans come pre-loaded with sociability and goodness goals. Those few who don't, we rightly regard as varelse. A sufficiently smart rogue general AI will indeed understand that it is doing wrong, harmful, senseless things. It just won't care.

You're not really following me.

If it's no more contemplative than a human, it's probably no more intelligent than a human. If it's no more intelligent than a human then it's not going to take over the world and fill it with paperclips.

Something capable of conquering the world or otherwise totally remaking reality in its image would by definition be something of vastly superior intelligence to ours. I find it hard to believe that this being wouldn't also be superior at self-reflection, since the two seem at least related.

If intelligence and self-awareness are unrelated phenomena, we have bigger problems. You could put that on a list of possible very depressing answers to the Fermi paradox: "the universe is full of non-sentient super-intelligences that are busy converting their planets' masses into rubber dog poop."

Re: Apocalypse soon: the scientists preparing for the end times

#50

Post by api »

JulianMorrison wrote: post #48
mariusz79 wrote: post #43

Earlier quoted context omitted.

I don't think anyone can debate that.. But it still does not explain why such an advanced being would continue producing paper clips when it's a. waste of it's time, b. waste of available resources. Yes the paper clip ai could decide that continuous production of paper clips does not help with it's own goals and decide that the paper clips will not be needed if there are no people on this planet and simply kill us al…

Let me put it this way. Why do humans continue to do all sorts of obviously stupid counterproductive things, at large and small scales? We know that reproducing so many of us is stupid. We know that eating so much fat, sugar and salt is stupid. We know that overheating the atmosphere is stupid. We know that destroying biodiversity, overfishing to the point of extinction, trashing the forests we need to breathe etc et…

I guess my assumption is that increased intelligence should bring with it increased self-awareness.

If it doesn't, then... well... like I said elsewhere in the thread... put that on your list of potential depressing answers to the Fermi paradox.

Post reply on HN