Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGI Doom and the Drake Equation

iamnotarobot.substack.com

Post reply on HN
Print view View original

31–40 of 113 posts

Re: AGI Doom and the Drake Equation

#31

Post by turtleyacht »

slibhb wrote: post #17
mrob wrote: post #13

Earlier quoted context omitted.

I think you're misinterpreting the argument. The paperclip maximizer scenario is not "over-optimizing" anything, it's an example of that same ethical pluralism you mention. The paperclip maximizer believes that maximizing paperclips is the highest possible good. There is only one set of facts about reality, and rationalism aims to find that set, but it makes no claims about what should be done with that information.…

> Human ethical norms are highly complex, and are the result of a long evolutionary history that will not be shared by any AI. The chances of any arbitrary set of ethical values being compatible with human life is very low. If AI is trained by a huge corpus of human language, it may very well share our norms/values.

Wouldn't it be split along language lines?

English, Russian, Spanish, etc.

I'm not sure how much the LLM has vacuumed up, or whether anyone has appended to their prompts, "in Portuguese."

It would be interesting to see how interpretations differ depending on the "translation," or if there is universal agreement.

Re: AGI Doom and the Drake Equation

#32

Post by nh23423fefe »

jacquesm wrote: post #10
nh23423fefe wrote: post #6

The nonexistence of grey goo (von neumann probes) is strong prior for safe agi. AI xrisk is woo. Paperclip maximizers are p-zombies. They can't exist. Chicken littles see apocalypses on every horizon even though they dont understand the technology at all. "I can imagine this destroying the world" is their justification. Even though their "imagination" is 16x16 greyscale.

The assumption that von Neumann probes would have made it here timely is not necessarily true, and if they had we wouldn't be having this conversation. So this doesn't really prove anything.

What is the content here? An observation isn't a truth? Who said it was?

Or are you trying to make some anthropic argument around survivorship bias.

Re: AGI Doom and the Drake Equation

#33

Post by d--b »

Noone predicted GPT. Even after AlphaGo, passing the Turing test was still a distant horizon. Passing the bar exam? Forget it! Everyone thought we’d have self driving cars before coding robots.

The lesson is we don’t fucking know what’s going to happen. Be humble people.

Re: AGI Doom and the Drake Equation

#34

Post by nh23423fefe »

blibble wrote: post #7
nh23423fefe wrote: post #6

The nonexistence of grey goo (von neumann probes) is strong prior for safe agi. AI xrisk is woo. Paperclip maximizers are p-zombies. They can't exist. Chicken littles see apocalypses on every horizon even though they dont understand the technology at all. "I can imagine this destroying the world" is their justification. Even though their "imagination" is 16x16 greyscale.

> The nonexistence of grey goo (von neumann probes) is strong prior for safe agi. what if we're first? or FTL travel isn't possible in our universe?

Sure, have a different prior if you think those assumptions are relevant. i don't

    P(what if we're first?) ~ 0

    P(or FTL travel isn't possible in our universe?) ~ 1

Re: AGI Doom and the Drake Equation

#35

Post by ctoth »

JumpCrisscross wrote: post #28
ctoth wrote: post #23

Earlier quoted context omitted.

> One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. So Aumann's Agreement Theorem[0]? > Implicit in this belief, it seems, is that there aren't really a naturally varying infinite set of values, or moral beliefs, that we all reason from. No, there probably aren't an infinity of priors with each person having a differe…

> most people who live in the US in 2023 believe that murder is bad, for instance Because we define away military conflict, the intentional taking of others’ lives.

As evidence against my idea that most people have similar ethical beliefs, I'm not sure what this is supposed to do other than win you a pedant point? So I upvoted you. But if you must, use rape instead of murder as your bad thing that most people believe is bad.

Re: AGI Doom and the Drake Equation

#36

Post by notahacker »

mrob wrote: post #13
tunesmith wrote: post #2

One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…

I think you're misinterpreting the argument. The paperclip maximizer scenario is not "over-optimizing" anything, it's an example of that same ethical pluralism you mention. The paperclip maximizer believes that maximizing paperclips is the highest possible good. There is only one set of facts about reality, and rationalism aims to find that set, but it makes no claims about what should be done with that information.…

The "paperclip maximiser" scenario is a scenario in which there is such an absence of ethical pluralism amongst AIs that they all unite to optimise paperclip production. (Or else that the paperclip manufacturing AIs are so vastly superior at strategy and resource to all other intelligences on a planet that they can defeat the combined forces of all the humans and AIs that don't want to be turned into paperclips)

Ethical pluralism implies that AIs don't all agree on a goal or even identify other AIs as having any positive value at all. Given hypothetical AIs with agency a lot of ability to exert force, this might still be problematic, but it's quite different from the popular movie unified AI vs humans scenario which seems to dominate "rationalist" discourse...

Re: AGI Doom and the Drake Equation

#37

Post by m3kw9 »

All the AGI must wipe out humanity theories are weird. Did we need to wipe out ants? Or anything to reign supreme on earth? Why would they wipe us out be the likely thing if they do reign supreme? Ha, I get it, we want to stay on top at all costs..

Re: AGI Doom and the Drake Equation

#38

Post by hiAndrewQuinn »

slibhb wrote: post #17
mrob wrote: post #13

Earlier quoted context omitted.

I think you're misinterpreting the argument. The paperclip maximizer scenario is not "over-optimizing" anything, it's an example of that same ethical pluralism you mention. The paperclip maximizer believes that maximizing paperclips is the highest possible good. There is only one set of facts about reality, and rationalism aims to find that set, but it makes no claims about what should be done with that information.…

> Human ethical norms are highly complex, and are the result of a long evolutionary history that will not be shared by any AI. The chances of any arbitrary set of ethical values being compatible with human life is very low. If AI is trained by a huge corpus of human language, it may very well share our norms/values.

Nobody wants an AI that shares norms and values with us, though. We just want a machine which does its job as efficiently as possible. Norms are brakes on actions which are fundamentally at odds with the nature of economic activity.

Re: AGI Doom and the Drake Equation

#39

Post by apples_oranges »

d--b wrote: post #33

Noone predicted GPT. Even after AlphaGo, passing the Turing test was still a distant horizon. Passing the bar exam? Forget it! Everyone thought we’d have self driving cars before coding robots. The lesson is we don’t fucking know what’s going to happen. Be humble people.

Actually you can draw some conclusions from what's going on. If iPhone 7 is faster than your Mac Book Pro, you may assume that one day Apple Silicon will be fast and low power consumption. If you see that suddenly machine translation makes a noticeable leap, you can assume that some progress was made in machine language processing etc.

And actually a lot of people predicted GPT and more, one notable person is probably Ray Kurzweil. Some others are all the folks at Open AI to set out to make it a reality.

Nono, "Noone predicted GPT" is just not true, I think.

Re: AGI Doom and the Drake Equation

#40

Post by tunesmith »

ctoth wrote: post #23
tunesmith wrote: post #2

One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. And that disagreement just means that one side is "wrong" and that therefore more debate is needed. This seems to be connected to the belief that AI will naturally just over-optimize to turn us all into paper clips. Implicit in this belief, it seems, is that there aren…

> One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. So Aumann's Agreement Theorem[0]? > Implicit in this belief, it seems, is that there aren't really a naturally varying infinite set of values, or moral beliefs, that we all reason from. No, there probably aren't an infinity of priors with each person having a differe…

One person may believe in maximizing the quality of current human lives. One may believe in maximizing the probability of future lives. One may believe in maximizing the health of the planet. They can all reason correctly and reach different normative conclusions. Aumann's makes no allowance for normative conclusions.
Post reply on HN