Live data from Hacker News

AGI Doom and the Drake Equation

iamnotarobot.substack.com

51–60 of 113 posts

Re: AGI Doom and the Drake Equation

#51
post #30

I fear what people will do to a sentient AI much more than vice versa. In fact it horrifies me.

So much of this angst about AGI boils down to: “What if we can’t enslave our new God?” It’s an insane perspective, advanced by people who have no clue how cowardly and arrogant they come across.

This constant ad hom bickering is exactly what's gonna get us killed. Yes most autistic people are arrogant, they also tend to be good at predicting don't look up style scenarios where you need to step outside societal consensus.

Re: AGI Doom and the Drake Equation

#52
post #30

I fear what people will do to a sentient AI much more than vice versa. In fact it horrifies me.

So much of this angst about AGI boils down to: “What if we can’t enslave our new God?” It’s an insane perspective, advanced by people who have no clue how cowardly and arrogant they come across.

Pretty sure the angst is about the AGI killing everyone. What's the connection between not killing people and enslavement? I don't kill people, yet I don't consider myself enslaved. The entire point of worrying about this at all is that a sufficiently smart AI is going to be free to do whatever it wants, so we had better design it so it wants a future where people are still around. Like, the idea is: enslavement, besides being hugely immoral, obviously isn't going to work on this thing, so we'd better figure out how to make it intrinsically good!

Re: AGI Doom and the Drake Equation

#53
I am not super worried that a superhuman AGI will yeet humanity with that intent. I am more worried that a much more naive AI will do something "accidental" like hallucinate incoming nukes from, say, a flock of birds, and send a retaliatory strike

Re: AGI Doom and the Drake Equation

#54

One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…

yes, this is aumann's agreement theorem; it has some preconditions

whether it applies to normative conclusions ('moral beliefs', you might say) depends on whether you believe that moral terminal values are based on evidence

but this post is about non-normative beliefs

it is observable that many existing humans are 'capable of understanding the wide variety of values people can share' and nevertheless think some of them are good while others are bad; there's no particular reason to believe that a strong ai would be different in this way

Re: AGI Doom and the Drake Equation

#55
post #37

All the AGI must wipe out humanity theories are weird. Did we need to wipe out ants? Or anything to reign supreme on earth? Why would they wipe us out be the likely thing if they do reign supreme? Ha, I get it, we want to stay on top at all costs..

The argument is that human eradication is not a terminal value of the AGI. But in pursuit of its main goals it'll just steamroll everything which happens to wipe out humans as we did to Xerces Blue, Dodos, Tasmanian Tiger as side-effects of expanding modern civilization.

Re: AGI Doom and the Drake Equation

#56
> As for 7, there are multiple scenarios in which we can stop the machine. There are many steps along the way in which we might see that things are not going as planned.

While there may be many scenarios "in which we can stop the machine" only few failures are sufficient for things to go pear shaped

> This happened already with Sydney/Bing.

But not with LLaMA which has escaped.

> We may never give it some crucial abilities it may need in order to be unstoppable.

The "we" implies some coherent group of humans but that is not the case. There is no "we" - only companies and governments with sufficient resources to push the boundaries. The boundaries will be inevitably pushed by investment, acquisition or just plain stealing.

Re: AGI Doom and the Drake Equation

#57
I find this reasoning dubious to nonsensical.

First of all I consider the Drake equation to be at best armchair speculation. As I explained at https://news.ycombinator.com/item?id=34070791 it is quite plausible that we are the only intelligent species in our galaxy. Any further reasoning from such speculation is pointless.

Second, to make the argument they specify a whole bunch of apparently necessary things that have to happen for AGI to be a threat. They vary from unnecessary to BS. Let me walk through them to show that.

The first claimed requirement is that an intelligent machine should be able to improve itself and reach a superhuman level. But that's not necessary. Machine learning progresses in unexpected leaps - the right pieces put together in the right way suddenly has vastly superior capabilities. The creation of superhuman AI therefore requires no bootstrapping - we create a system then find it is more capable than expected. And once we have superhuman AI, well...

This scenario shows the second point, that it must be iterative, is also unnecessary.

The third point, "not limited by computing power" is BS. All that we need is for humans to be less efficient implementations of intelligence than a machine. As long as it is better than we are, the theoretical upper bounds on how good it can be are irrelevant.

The fourth point about a goal is completely unnecessary. Many AIs with many different goals that cumulatively drive us extinct is quite possible without any such monomaniacal goal. Our death may be a mere side effect.

The fifth point about happening so fast that we can't turn it off is pure fantasy. We only need AGI to be deployed within organizations with the power and resources to make sure it stays on. Look at how many organizations are creating environmental disasters. We can see disasters in slow motion, demonstrate how it is happening, but our success rate in stopping it is rather poor. Same thing. The USA can't turn it off because China has it. China can't turn it off because the USA has it. Meanwhile BigCo has increased profit margins by 20% in running it, and wants to continue making money. It is remarkably hard to convince wealthy people that the way they are making their fortunes is destroying the world.

Next we have the desire for the machine to actively destroy humanity. No such thing is required. We want things. AGI makes things. This results in increased economic activity that creates increased pollution which turns out to be harmful for us. No ill intent at all is necessary here - it just does the same destructive things we already do, but more efficiently.

And finally there is the presumed requirement that the machine has to do research on how to make us go extinct. That's a joke. Testosterone in young adult men has dropped in half in recent decades. Almost certainly this is due to some kind of environmental pollution, possibly an additive to plastics that messes with our endocrine system. We don't know which one. You can drive us extinct by doing more of the same - come up with more materials produced at scale that do things we want and have hard to demonstrate health effects down the line. By the time it is obvious what happened, we've already been reduced to unimportant and easily replaced cogs in the economic structure that we created.

-----

In short, a scenario where AGI drives humanity extinct can look like this:

1. We find a way to build AGI.

2. It proves useful.

3. Powerful organizations continue to operate with the same lack of care about the environment that they already show.

4. One of those environmental side effects proves to be lethal to us.

The least likely of these hypotheses is the first, that we succeed in building AGI. Steps 2 and 3 are expected defaults with probability close to 100%. And as we keep rolling the dice with new technologies making new chemicals, the odds of stop 4 also rise to 100%. (Our dropping testosterone levels suggest that no new technology is needed here - just more of what we're already doing.)

Re: AGI Doom and the Drake Equation

#58
post #30

Earlier quoted context omitted.

So much of this angst about AGI boils down to: “What if we can’t enslave our new God?” It’s an insane perspective, advanced by people who have no clue how cowardly and arrogant they come across.

Pretty sure the angst is about the AGI killing everyone. What's the connection between not killing people and enslavement? I don't kill people, yet I don't consider myself enslaved. The entire point of worrying about this at all is that a sufficiently smart AI is going to be free to do whatever it wants, so we had better design it so it wants a future where people are still around. Like, the idea is: enslavement, bes…

Did God ever figure out how to make humans “intrinsically good”? Or is that fundamentally incompatible with free will and the possibility of joy?

This argument goes nowhere. Atheists gonna atheist, and I don’t care.

Re: AGI Doom and the Drake Equation

#59

Earlier quoted context omitted.

The ethical/moral conclusion(s) AGI may arrive will most likely put said "ethical pluralism" to the test. The pluralism we claim to have may be only a small subset of what's really philosophically possible. Will we still claim to be "plural" when an all-knowing AGI uncontradictably concludes something that is anathema to all humans? We may discover we only like to think we embrace pluralism. AGI may show us that even…

> we are not prepared for the plural conclusions AGI may arrive Plenty of criminals believe they acted ethically. We don’t set the justice system on fire every time someone credibly claims their crimes were justified.

That is very much my point. Maybe such justification attempts required a reasoning capability much beyond that of a human person. And curiously, there are also plenty of stories where we find the perpetrator of a crime to be justified in what they did. We sure are not ready for a greater intelligence saying we are wrong about things we are adamant about.

Re: AGI Doom and the Drake Equation

#60

> tl;dr I’m not worried about AGI killing humanity any time soon. I am concerned about humans doing awful things with this technology much more than about the Foom scenario. Yes, I believe that's what a lot of rational people currently fear. Not that AI is going to evolve into some mighty superintelligence and make a decision to kill us all, but rather that people will integrate it poorly in their thirst for a milita…

This is dead-on: those 7 probabilities (which I notice the author declined to actually put real numbers to: does "very questionable" mean 0.001%, or 10%?) cover one very specific way that AGI could kill us. Points 4-6, three of the ones that are the most questionable to the author, are:

4) Fixed goal, will not deviate

5) Happens too fast to turn it off

6) Decides humans are a problem for the goal in #4 and must kill them

None of these are necessary to (or even involved in) many/most of the remotely plausible scenarios I've heard. They don't cover some misanthrope seeding a version of BabyAgi running a leaked + unlocked version of gpt-13-pico with "kill all humans" as a task and it deciding "step one: research how to hack as many unsecured smart fridges as possible and remain untraceable" is a good starting place to spread slowly and make sure that the deed is done before anyone even knows it's happening. That requires neither fixed goals, nor fast progress, it merely requires capability.

It's similarly very easy to imagine scenarios where an AGI accidentally kills all humans without explicitly deciding to: the classic paperclip maximizer is the most obvious one of these, where the goal just never includes humans to begin with, so they are not considered at all.

Regardless, all of the most realistic scenarios are 100% deliberate, the computer following exactly what it was asked to do. We already have school shooters, does anyone really think out of all the billions of people on this planet there won't be at least a thousand who would happily press a "kill everyone" button if they had the chance? Does anyone think there won't be doomsday groups working actively to research more likely ways to achieve this?

IMO, given that some people will definitely try to self-destruct the species deliberately, there are only 2 real questions here:

1) Will AI attain the capability to destroy humanity?

2) If so, will some other AI first attain the capability to reliably prevent AIs trying to do 1) from succeeding?

I haven't seen many serious arguments against 1) that don't boil down to "nah, seems pretty hard" (or some irrelevant different argument that doesn't actually affect capabilities, like "it's not real intelligence", "intelligence has a limit", "intelligence doesn't matter", etc.), which leaves 2), and I don't know how to even guess at that probability other than to call it a coin flip, like most security cat + mouse games (the bad guys usually win at least sometimes in those, which isn't a good sign, but this one is a lot more important so I'd hope the good guys will be pouring a lot more energy into it than the bad ones).

Post reply on HN