Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGI Doom and the Drake Equation

iamnotarobot.substack.com

Post reply on HN
Print view View original

81–90 of 113 posts

Re: AGI Doom and the Drake Equation

#81

Post by swombat »

slibhb wrote: post #17
mrob wrote: post #13

Earlier quoted context omitted.

I think you're misinterpreting the argument. The paperclip maximizer scenario is not "over-optimizing" anything, it's an example of that same ethical pluralism you mention. The paperclip maximizer believes that maximizing paperclips is the highest possible good. There is only one set of facts about reality, and rationalism aims to find that set, but it makes no claims about what should be done with that information.…

> Human ethical norms are highly complex, and are the result of a long evolutionary history that will not be shared by any AI. The chances of any arbitrary set of ethical values being compatible with human life is very low. If AI is trained by a huge corpus of human language, it may very well share our norms/values.

Our norms and values include that we treat sentient creatures that we deem inferior as if they had no moral value.

So that's not very comforting tbh.

Re: AGI Doom and the Drake Equation

#82

Post by diego »

c1ccccc1 wrote: post #11

Weird. Seems like this is a plagiarism of this article: https://www.strangeloopcanon.com/p/agi-strange-equation (Or maybe it's an independent reinvention of the same idea, and something's just in the water.)

I had not seen that article. I'm not surprised someone had a similar idea before.

Re: AGI Doom and the Drake Equation

#83

Post by btilly »

lostmsu wrote: post #76
btilly wrote: post #66

Earlier quoted context omitted.

I find it hilarious that rationalists have failed to notice or realize the consequences of the fact that approximating an update to a Bayesian network, even to getting an approximate probability answer that is within 49% of the real one, is NP hard. The consequence is that for any moderately complex set of beliefs, it is computationally impossible for us to reason hard enough about any particular observation to corre…

Point is not reasoning brings worse outcomes. It doesn't have to be perfect.

That is an argument for learning how to reason. Which is not actually the point under discussion. Going back to the parent of my comment:

> One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions.

What I'm showing is that results like https://www.sciencedirect.com/science/article/abs/pii/000437... demonstrate that this belief is incorrect. Two people starting with the same priors, same observations, and same views on rationality may do the best they can and come to diametrically opposed conclusions. And a lifetime of discussion may be too little to determine which one is right. Ditto putting all the computers in the world to work on the problem for a lifetime.

Real life is worse. We start with different priors and our experiences include different observations. Which makes stark disagreements even easier than when you start with an ideal situation of identical priors and experiences.

This result should encourage us to have humility about the certainty which can be achieved through rationality. But few rationalists show anything like that form of humility.

Re: AGI Doom and the Drake Equation

#84

Post by notahacker »

mrob wrote: post #41
notahacker wrote: post #36

Earlier quoted context omitted.

The "paperclip maximiser" scenario is a scenario in which there is such an absence of ethical pluralism amongst AIs that they all unite to optimise paperclip production. (Or else that the paperclip manufacturing AIs are so vastly superior at strategy and resource to all other intelligences on a planet that they can defeat the combined forces of all the humans and AIs that don't want to be turned into paperclips) Ethi…

The paperclip maximizer scenario assumes that recursive self-improvement is possible, which means there will most likely be only a single AI of superhuman power.

There are ancillary assumptions implicit in that (either recursive self improvement and the decision to eliminate humanity is so fast that no counter intelligence uninterested in paperclips can be made, or that how recursive self improvement has been achieved is such a mystery that no counter intelligence uninterested in paperclips can be made. Also that recursive self improvement doesn't itself involve either sequentially or simultaneously coming to adopt a range of views on the value of humans with respect to paperclips)

To be fair, assumptions about single superhuman intelligences make a little more sense if we're talking about a secret Skynet project carried out by a state's most advanced research labs and not a mundane little office supplier's program accidentally achieving the singularity after being tweaked for paperclip output.

Re: AGI Doom and the Drake Equation

#85

Post by robwwilliams »

btilly wrote: post #57

I find this reasoning dubious to nonsensical. First of all I consider the Drake equation to be at best armchair speculation. As I explained at https://news.ycombinator.com/item?id=34070791 it is quite plausible that we are the only intelligent species in our galaxy. Any further reasoning from such speculation is pointless. Second, to make the argument they specify a whole bunch of apparently necessary things that hav…

I disagree with most of the assumptions that get you close to P = 1. But I enjoyed your comment anyway. It makes me examine my own assignment of probabilities. I gave 1 and 2 high Ps and the others rather low or undefinable Ps.

Re: AGI Doom and the Drake Equation

#86

Post by d--b »

apples_oranges wrote: post #39
d--b wrote: post #33

Noone predicted GPT. Even after AlphaGo, passing the Turing test was still a distant horizon. Passing the bar exam? Forget it! Everyone thought we’d have self driving cars before coding robots. The lesson is we don’t fucking know what’s going to happen. Be humble people.

Actually you can draw some conclusions from what's going on. If iPhone 7 is faster than your Mac Book Pro, you may assume that one day Apple Silicon will be fast and low power consumption. If you see that suddenly machine translation makes a noticeable leap, you can assume that some progress was made in machine language processing etc. And actually a lot of people predicted GPT and more, one notable person is probabl…

In 2015? They knew they were going to make a chat bot. But they didn’t dream it would be THAT strong

Re: AGI Doom and the Drake Equation

#87

Post by robwwilliams »

mrob wrote: post #80
robwwilliams wrote: post #75

Earlier quoted context omitted.

I do not think so. If a true AGI were to select its own version of meaning (paperclips or marbles) would it not select something along the lines of “more knowledge of the universe in which I find myself”? It is presumably going to have superintelligence, so let’s give it/them a better and more plausible meaning; something other than paperclips, marbles, or von Neumann machines.

"Knowledge of the universe" is just as dangerous a terminal goal as "maximize paperclips". To paraphrase Yudkowsky, you are made from atoms, which could be used to build the super-ultra-large particle collider.

No, I disagree. More knowledge equals more diverse dynamic structures and interactions—more “good” entropy. That covaries with higher diversity, not more paperclips.

Re: AGI Doom and the Drake Equation

#88

Post by btilly »

robwwilliams wrote: post #85
btilly wrote: post #57

I find this reasoning dubious to nonsensical. First of all I consider the Drake equation to be at best armchair speculation. As I explained at https://news.ycombinator.com/item?id=34070791 it is quite plausible that we are the only intelligent species in our galaxy. Any further reasoning from such speculation is pointless. Second, to make the argument they specify a whole bunch of apparently necessary things that hav…

I disagree with most of the assumptions that get you close to P = 1. But I enjoyed your comment anyway. It makes me examine my own assignment of probabilities. I gave 1 and 2 high Ps and the others rather low or undefinable Ps.

I'm curious how you get rather low or undefinable Ps for the other two.

Look into the history of how the tobacco industry tried to suppress research on the harms of smoking, how the sugar industry tried to shift blame for health problesm like obesity from sugars to fats, and how the fossil fuel industry has resisted attempts to take responsibility for global warming. In all three cases not only did industry resist evidence of harm, it also funded publicity campaigns to try to shift public opinion their way.

Given that I know of no reason to believe that this will change, I think that point 3 has high probability.

As for point 4, we have a history of spreading chemicals widely before discovering bad things about them. The first such chemical to gain notoriety was DDT, but many more have followed. In the last few decades, https://www.pnas.org/doi/10.1073/pnas.2023989118 shows that flying insect biomass dropped by 3/4. https://www.urologytimes.com/view/testosterone-levels-show-s... likewise shows that, even after controlling for known factors like increased obesity, there is an unexplained decline in testosterone of roughly 1/3. It is reasonable to guess that both are the result of environmental factors. But we are not sure what factors those are, and are not significantly modifying our behavior. (How could we, when we don't know for sure what we are doing to cause the problem?)

Given these examples, I truly believe we are rolling environmental dice with our health. And if we keep rolling the dice, eventually we'll come up snake eyes.

Re: AGI Doom and the Drake Equation

#90

Post by blibble »

nh23423fefe wrote: post #34
blibble wrote: post #7

Earlier quoted context omitted.

> The nonexistence of grey goo (von neumann probes) is strong prior for safe agi. what if we're first? or FTL travel isn't possible in our universe?

Sure, have a different prior if you think those assumptions are relevant. i don't P(what if we're first?) ~ 0 P(or FTL travel isn't possible in our universe?) ~ 1

what's your basis for these assumptions?
Post reply on HN