Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGI Doom and the Drake Equation

iamnotarobot.substack.com

Post reply on HN
Print view View original

21–30 of 113 posts

Re: AGI Doom and the Drake Equation

#21

Post by ActorNightly »

There is even a simpler explanation.

In order for AGI to even begin, it needs to self develop a method to improve itself.

That means that the initial code that runs has to end up producing something that looks like an inference->error->training loop without any semblance of that being in the original code.

No system in existence can do that, nor do we even have any idea of what that may even look like.

The closest that we will get to AGI would be equivalent of a very smart human, who can still very much be controlled.

Re: AGI Doom and the Drake Equation

#22

Post by mrob »

tunesmith wrote: post #16
mrob wrote: post #13

Earlier quoted context omitted.

I think you're misinterpreting the argument. The paperclip maximizer scenario is not "over-optimizing" anything, it's an example of that same ethical pluralism you mention. The paperclip maximizer believes that maximizing paperclips is the highest possible good. There is only one set of facts about reality, and rationalism aims to find that set, but it makes no claims about what should be done with that information.…

Implicit in creating paper clips is its belief that it should create paper clips, which is a normative conclusion.

Rationalism does not claim that any entity should maximize paperclips, only that such an ethical norm could exist. And if something vastly more powerful than humans has that ethical norm, things will end very badly for us.

Re: AGI Doom and the Drake Equation

#23

Post by ctoth »

tunesmith wrote: post #2

One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…

> One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions.

So Aumann's Agreement Theorem[0]?

> Implicit in this belief, it seems, is that there aren't really a naturally varying infinite set of values, or moral beliefs, that we all reason from.

No, there probably aren't an infinity of priors with each person having a different one. Probably most people who live in the US in 2023 believe that murder is bad, for instance.

And because "ethical pluralism" or rather, some people will want to murder, AGI won't kill us?

Not really sure how this is all supposed to work but it sounds a little less developed of a "not kill everybody" plan than the rationalists have.

> But the end state of a system that is capable of understanding the wide variety of values people can share isn't exactly going to take a stand on any particular set of values unless instructed to.

Why not?

[0]: https://www.lesswrong.com/tag/aumann-s-agreement-theorem

Re: AGI Doom and the Drake Equation

#24

Post by progrus »

excalibur wrote: post #4

> tl;dr I’m not worried about AGI killing humanity any time soon. I am concerned about humans doing awful things with this technology much more than about the Foom scenario. Yes, I believe that's what a lot of rational people currently fear. Not that AI is going to evolve into some mighty superintelligence and make a decision to kill us all, but rather that people will integrate it poorly in their thirst for a milita…

There is a clear distinction. In one group are the Yud cult, who are rediscovering their fear of God, and pretending that it is an intellectual exercise. In the other group are people who see risks with databases, facial recognition, “weak” AI and the rest. I’m in that group, but still skeptical of basically all doom stories.

Re: AGI Doom and the Drake Equation

#26

Post by JumpCrisscross »

jacquesm wrote: post #8
tunesmith wrote: post #2

One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…

Interesting observation. And yet, societies have many mechanisms to reduce the amount of ethical pluralism such as laws, conventions & customs, peer pressure, religions and so on. It seems as though we will tolerate some ethical pluralism but not too much of it. There is this 'bandwidth' of acceptable behavior and if you go too far out of it bad stuff will happen to you: you get ostracized, put in jail, a psychiatric…

> societies have many mechanisms to reduce the amount of ethical pluralism such as laws, conventions & customs, peer pressure, religions and so on

Constrain, yes. The same way we would seek to constrain a paperclip-maximising LLM.

Re: AGI Doom and the Drake Equation

#27

Post by camel-cdr »

just some thoughts on some of the requirements:

> 3. This improvement is not limited by computing power, or at least not limited enough by the computing resources and energy available to the substrate of the machine.

While this is a requirement, this doesn't mean that the points 4, 6 and 7 apply to the same, let's call it, generation of the AI that "escaped" from a resource limited server. There may not even be a self improval before an unnoticed "escape".

> 4. This system will have a goal that it will optimize for, and that it will not deviate from under any circumstances regardless of how intelligent it is. If the system was designed to maximize the number of marbles in the universe, the fact that it’s making itself recursively more intelligent won’t cause it to ever deviate from this simple goal.

I don't see how that is a requirement. The last sentence seems to imply that deviating from the initial optimization goals automatically means the AI developed morals, and/or we don't have to worry. But I don't see any reason to believe that.

> 5. This needs to happen so fast that we cannot turn it off (also known as the Foom scenario).

Well, that, or it could also happen slow and gradually, but stay unnoticed.

> 7. It’s possible for this machine to do the required scientific research and build the mechanisms to eliminate humanity before we can defend ourselves and before we can stop it.

... or before we notice.

Re: AGI Doom and the Drake Equation

#28

Post by JumpCrisscross »

ctoth wrote: post #23
tunesmith wrote: post #2

One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…

> One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. So Aumann's Agreement Theorem[0]? > Implicit in this belief, it seems, is that there aren't really a naturally varying infinite set of values, or moral beliefs, that we all reason from. No, there probably aren't an infinity of priors with each person having a differe…

> most people who live in the US in 2023 believe that murder is bad, for instance

Because we define away military conflict, the intentional taking of others’ lives.

Re: AGI Doom and the Drake Equation

#29

Post by mrob »

slibhb wrote: post #17
mrob wrote: post #13

Earlier quoted context omitted.

I think you're misinterpreting the argument. The paperclip maximizer scenario is not "over-optimizing" anything, it's an example of that same ethical pluralism you mention. The paperclip maximizer believes that maximizing paperclips is the highest possible good. There is only one set of facts about reality, and rationalism aims to find that set, but it makes no claims about what should be done with that information.…

> Human ethical norms are highly complex, and are the result of a long evolutionary history that will not be shared by any AI. The chances of any arbitrary set of ethical values being compatible with human life is very low. If AI is trained by a huge corpus of human language, it may very well share our norms/values.

I don't think that's likely, because human values came into existence because of the evolved goal of reproductive fitness in an environment of natural selection. LLMs are trained to imitate human language, not to have many descendants. They could well "understand" human language (in whatever meaning you choose to interpret that), but that doesn't mean that imitating human values is the mechanism by which they will do so. The success of current LLMs suggests that there's a much simpler way to do it.

Re: AGI Doom and the Drake Equation

#30

Post by progrus »

kerblang wrote: post #25

I fear what people will do to a sentient AI much more than vice versa. In fact it horrifies me.

So much of this angst about AGI boils down to: “What if we can’t enslave our new God?”

It’s an insane perspective, advanced by people who have no clue how cowardly and arrogant they come across.

Post reply on HN