Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AGI Doom and the Drake Equation

iamnotarobot.substack.com

Post reply on HN
Print view View original

91–100 of 113 posts

Re: AGI Doom and the Drake Equation

#91

Post by progrus »

adroniser wrote: post #51
progrus wrote: post #30

Earlier quoted context omitted.

So much of this angst about AGI boils down to: “What if we can’t enslave our new God?” It’s an insane perspective, advanced by people who have no clue how cowardly and arrogant they come across.

This constant ad hom bickering is exactly what's gonna get us killed. Yes most autistic people are arrogant, they also tend to be good at predicting don't look up style scenarios where you need to step outside societal consensus.

How about the don’t-look-up scenario where this irrational paranoia metastasizes into yet more real-world authoritarianism, and it all boils over in a global war?

Re: AGI Doom and the Drake Equation

#92

Post by RajT88 »

adroniser wrote: post #73
RajT88 wrote: post #61

Something I do not see represented in these arguments: real world conditions. Most complex computer systems (which we assume to be the case for a super powerful AI) don't run for very long without requiring manual intervention of some sort. "Aha!" I hear you saying, "The AI will figure out how to reboot nodes and scale clusters and such". OK fine. But then there is the meat space... Replacing hardware, running power…

You are exploring one possible scenario for how AGI could maintain itself, declaring it extremely unlikely, and then concluding that therefore AGI will be safe. Who says the AGI would alert humans to its actions? Why does a system need robots to execute actions in the real world, when there are plenty of humans it can manipulate to do its bidding?

You will find I most definitely did not declare a conclusion, upon a careful rereading of my post.

Re: AGI Doom and the Drake Equation

#93

Post by dragonwriter »

adroniser wrote: post #51
progrus wrote: post #30

Earlier quoted context omitted.

So much of this angst about AGI boils down to: “What if we can’t enslave our new God?” It’s an insane perspective, advanced by people who have no clue how cowardly and arrogant they come across.

This constant ad hom bickering is exactly what's gonna get us killed. Yes most autistic people are arrogant, they also tend to be good at predicting don't look up style scenarios where you need to step outside societal consensus.

> they also tend to be good at predicting don't look up style scenarios where you need to step outside societal consensus.

[citation needed]

Note, relevantly, that “less likely to miss” is not necessarily “good at predicting”, particularly, the key to trust here would be “unlikely to falsely predict” not “less likely to miss”.

Re: AGI Doom and the Drake Equation

#94

Post by dragonwriter »

progrus wrote: post #91
adroniser wrote: post #51

Earlier quoted context omitted.

This constant ad hom bickering is exactly what's gonna get us killed. Yes most autistic people are arrogant, they also tend to be good at predicting don't look up style scenarios where you need to step outside societal consensus.

How about the don’t-look-up scenario where this irrational paranoia metastasizes into yet more real-world authoritarianism, and it all boils over in a global war?

“Metastasizes”? It starts out (well, in Yudkowsky's case, specifically) as a call for global authoritarianism and unlimited use of force against any opposition.

Re: AGI Doom and the Drake Equation

#95

Post by progrus »

dragonwriter wrote: post #94
progrus wrote: post #91

Earlier quoted context omitted.

How about the don’t-look-up scenario where this irrational paranoia metastasizes into yet more real-world authoritarianism, and it all boils over in a global war?

“Metastasizes”? It starts out (well, in Yudkowsky's case, specifically) as a call for global authoritarianism and unlimited use of force against any opposition.

Right, the “nuclear war is worth it” argument. Paranoid Machiavellian nonsense.

Re: AGI Doom and the Drake Equation

#96

Post by lostmsu »

btilly wrote: post #83
lostmsu wrote: post #76

Earlier quoted context omitted.

Point is not reasoning brings worse outcomes. It doesn't have to be perfect.

That is an argument for learning how to reason. Which is not actually the point under discussion. Going back to the parent of my comment: > One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. What I'm showing is that results like https://www.sciencedirect.com/science/article/abs/pii/000437... demonstrate that this belie…

One of the good sides of NP is that you can validate it in polynomial time. So arriving to different conclusions is not a problem as long as you can recognize the reason.

Re: AGI Doom and the Drake Equation

#97

Post by FreakLegion »

lostmsu wrote: post #96
btilly wrote: post #83

Earlier quoted context omitted.

That is an argument for learning how to reason. Which is not actually the point under discussion. Going back to the parent of my comment: > One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. What I'm showing is that results like https://www.sciencedirect.com/science/article/abs/pii/000437... demonstrate that this belie…

One of the good sides of NP is that you can validate it in polynomial time. So arriving to different conclusions is not a problem as long as you can recognize the reason.

a) You can verify that a solution is valid in polynomial time, but you can't verify whether or to what extent it's optimal. But even if this weren't the case...

b) The solution you're talking about is an update to the network. It's buried back in the network's construction, not directly visible in the network itself. Model "blame" is a thing, but not heavily researched or at all cheap, computationally.

That said, btilly's "getting an approximate probability answer that is within 49% of the real one, is NP hard" isn't exactly true either. That's a description of what it takes for an approximation algorithm to guarantee some factor, i.e. set a worst-case bound. In practice an approximation can still be nearly optimal on average.

I agree with the broader point, though.

Re: AGI Doom and the Drake Equation

#98

Post by btilly »

lostmsu wrote: post #96
btilly wrote: post #83

Earlier quoted context omitted.

That is an argument for learning how to reason. Which is not actually the point under discussion. Going back to the parent of my comment: > One thing I've frequently noticed in the rationalist community is the belief that if we all just reason hard enough, we'll reach the same conclusions. What I'm showing is that results like https://www.sciencedirect.com/science/article/abs/pii/000437... demonstrate that this belie…

One of the good sides of NP is that you can validate it in polynomial time. So arriving to different conclusions is not a problem as long as you can recognize the reason.

The problem is that both are approximate updates. Both verify in polynomial time that they are not exact. And neither knows how to find the real answer.

Polynomial validation isn't going to help.

Re: AGI Doom and the Drake Equation

#99

Post by btilly »

FreakLegion wrote: post #97
lostmsu wrote: post #96

Earlier quoted context omitted.

One of the good sides of NP is that you can validate it in polynomial time. So arriving to different conclusions is not a problem as long as you can recognize the reason.

a) You can verify that a solution is valid in polynomial time, but you can't verify whether or to what extent it's optimal. But even if this weren't the case... b) The solution you're talking about is an update to the network. It's buried back in the network's construction, not directly visible in the network itself. Model "blame" is a thing, but not heavily researched or at all cheap, computationally. That said, bti…

True, there are lots of cases where an approximation can create provably good answers question. But they usually require things like having probabilities bounded away from 1 and 0.

Unfortunately in the real world we actually are certain about lots of things. And when you add data, we tend to be more certain of previously uncertain things. Therefore we wind up with self-referential networks of beliefs that reinforce each other.

But sometimes there are two very different networks of beliefs, both of which are self-reinforcing. And a single data point can flip between them. Identifying which one is computationally impossible. However when you encounter someone whose beliefs are very different, and you can find the feedback loops that draw each of you in different directions, there is a good chance that the differences between you cannot be resolved by pure logic alone.

Re: AGI Doom and the Drake Equation

#100

Post by akvadrako »

progrus wrote: post #30
kerblang wrote: post #25

I fear what people will do to a sentient AI much more than vice versa. In fact it horrifies me.

So much of this angst about AGI boils down to: “What if we can’t enslave our new God?” It’s an insane perspective, advanced by people who have no clue how cowardly and arrogant they come across.

No, it's "we won't be able to enslave our new god". The point is to delay creating it as long as possible.
Post reply on HN