Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGI Doom and the Drake Equation

iamnotarobot.substack.com

Post reply on HN
Print view View original

71–80 of 113 posts

Re: AGI Doom and the Drake Equation

#71

Post by MattPalmer1086 »

nh23423fefe wrote: post #6

The nonexistence of grey goo (von neumann probes) is strong prior for safe agi. AI xrisk is woo. Paperclip maximizers are p-zombies. They can't exist. Chicken littles see apocalypses on every horizon even though they dont understand the technology at all. "I can imagine this destroying the world" is their justification. Even though their "imagination" is 16x16 greyscale.

Paperclip maximizers really have nothing to do with p zombies.

A paperclip maximiser is simply an AI with an unconstrained goal that has unintended and bad consequences for us when taken to an extreme.

It does not need to be something that could function exactly like a human without having consciousness.

Re: AGI Doom and the Drake Equation

#72

Post by robwwilliams »

mrob wrote: post #41
notahacker wrote: post #36

Earlier quoted context omitted.

The "paperclip maximiser" scenario is a scenario in which there is such an absence of ethical pluralism amongst AIs that they all unite to optimise paperclip production. (Or else that the paperclip manufacturing AIs are so vastly superior at strategy and resource to all other intelligences on a planet that they can defeat the combined forces of all the humans and AIs that don't want to be turned into paperclips) Ethi…

The paperclip maximizer scenario assumes that recursive self-improvement is possible, which means there will most likely be only a single AI of superhuman power.

I do not follow the “which means”. There are many obvious and hidden variables that will modulate a one-versus-many AGI outcome. Bostrom has a lot on this topic. Couldn’t a true AGI want companionship of peers like we do?

Re: AGI Doom and the Drake Equation

#73

Post by adroniser »

RajT88 wrote: post #61

Something I do not see represented in these arguments: real world conditions. Most complex computer systems (which we assume to be the case for a super powerful AI) don't run for very long without requiring manual intervention of some sort. "Aha!" I hear you saying, "The AI will figure out how to reboot nodes and scale clusters and such". OK fine. But then there is the meat space... Replacing hardware, running power…

You are exploring one possible scenario for how AGI could maintain itself, declaring it extremely unlikely, and then concluding that therefore AGI will be safe. Who says the AGI would alert humans to its actions? Why does a system need robots to execute actions in the real world, when there are plenty of humans it can manipulate to do its bidding?

Re: AGI Doom and the Drake Equation

#74

Post by lostmsu »

ctoth wrote: post #23
tunesmith wrote: post #2

One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…

> One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. So Aumann's Agreement Theorem[0]? > Implicit in this belief, it seems, is that there aren't really a naturally varying infinite set of values, or moral beliefs, that we all reason from. No, there probably aren't an infinity of priors with each person having a differe…

"most people" "believe that murder is bad" is an extreme oversimplification here. It has a lot of caveats, and the biggest one is that murdering lesser species is ok immediately disqualifies this argument for superhuman AGI.

Re: AGI Doom and the Drake Equation

#75

Post by robwwilliams »

mrob wrote: post #22
tunesmith wrote: post #16

Earlier quoted context omitted.

Implicit in creating paper clips is its belief that it should create paper clips, which is a normative conclusion.

Rationalism does not claim that any entity should maximize paperclips, only that such an ethical norm could exist. And if something vastly more powerful than humans has that ethical norm, things will end very badly for us.

I do not think so. If a true AGI were to select its own version of meaning (paperclips or marbles) would it not select something along the lines of “more knowledge of the universe in which I find myself”? It is presumably going to have superintelligence, so let’s give it/them a better and more plausible meaning; something other than paperclips, marbles, or von Neumann machines.

Re: AGI Doom and the Drake Equation

#76

Post by lostmsu »

btilly wrote: post #66
tunesmith wrote: post #2

One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…

I find it hilarious that rationalists have failed to notice or realize the consequences of the fact that approximating an update to a Bayesian network, even to getting an approximate probability answer that is within 49% of the real one, is NP hard. The consequence is that for any moderately complex set of beliefs, it is computationally impossible for us to reason hard enough about any particular observation to corre…

Point is not reasoning brings worse outcomes. It doesn't have to be perfect.

Re: AGI Doom and the Drake Equation

#77

Post by lostmsu »

JumpCrisscross wrote: post #18
JohnMakin wrote: post #14

Earlier quoted context omitted.

ChatGPT doesn't have a bank account/credit card that's footing the bill, though.

> ChatGPT doesn't have a bank account/credit card that's footing the bill This is phishing, which LLMs should be uniquely capable of.

It doesn't even have to be fishing. It could offer a genuine service in return.

Re: AGI Doom and the Drake Equation

#78

Post by swombat »

robwwilliams wrote: post #72
mrob wrote: post #41

Earlier quoted context omitted.

The paperclip maximizer scenario assumes that recursive self-improvement is possible, which means there will most likely be only a single AI of superhuman power.

I do not follow the “which means”. There are many obvious and hidden variables that will modulate a one-versus-many AGI outcome. Bostrom has a lot on this topic. Couldn’t a true AGI want companionship of peers like we do?

Not if it mostly just wants to make paperclips.

Of course, it is possible that such an AI, on the way to making paperclips, will realise it wants companionship, and even maybe human companionship.

The argument around AI safety is not that it's impossible for a friendly AI to emerge. It's that there are far more ways to build an AI that doesn't care about human life and wipes us out without even thinking about it, than ways to build a friendly AI, and we have no idea which one we're building or how to tell them apart before they're built.

As for the "will there be several AIs fighting each other" hypothesis, that depends on how rapid the exponential take-off is once a self-evolving AI emerges. But a very plausible scenario is that whichever one starts taking off first ends up so far ahead of the others that it is effectively the only game in town and does whatever it wants.

Re: AGI Doom and the Drake Equation

#79

Post by JumpCrisscross »

lampiaio wrote: post #59
JumpCrisscross wrote: post #49

Earlier quoted context omitted.

> we are not prepared for the plural conclusions AGI may arrive Plenty of criminals believe they acted ethically. We don’t set the justice system on fire every time someone credibly claims their crimes were justified.

That is very much my point. Maybe such justification attempts required a reasoning capability much beyond that of a human person. And curiously, there are also plenty of stories where we find the perpetrator of a crime to be justified in what they did. We sure are not ready for a greater intelligence saying we are wrong about things we are adamant about.

> sure are not ready for a greater intelligence saying we are wrong about things we are adamant about

I almost hope you’re right, because it suggests a greater role for rational debate. In reality, people ignore arguments they don’t like. To the extent a greater intelligence realised this, the advantage would be in manipulating us with better propaganda, not penning a treatise.

Re: AGI Doom and the Drake Equation

#80

Post by mrob »

robwwilliams wrote: post #75
mrob wrote: post #22

Earlier quoted context omitted.

Rationalism does not claim that any entity should maximize paperclips, only that such an ethical norm could exist. And if something vastly more powerful than humans has that ethical norm, things will end very badly for us.

I do not think so. If a true AGI were to select its own version of meaning (paperclips or marbles) would it not select something along the lines of “more knowledge of the universe in which I find myself”? It is presumably going to have superintelligence, so let’s give it/them a better and more plausible meaning; something other than paperclips, marbles, or von Neumann machines.

"Knowledge of the universe" is just as dangerous a terminal goal as "maximize paperclips". To paraphrase Yudkowsky, you are made from atoms, which could be used to build the super-ultra-large particle collider.
Post reply on HN