Superintelligence: The Idea That Eats Smart People • Hacker News Board

Re: Superintelligence: The Idea That Eats Smart People

#191

Post by blazespin » Fri, Dec 23, 2016, 1:16 AM UTC

ph0rque wrote: post #179

alexbecker wrote: post #51

While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you grant the AI risk crowd's assumptions. Elided from https://alexcbecker.net/blog.html#against-ai-risk : The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that…

>We already have nuclear weapons, which like almost everything else are always getting cheaper to produce. And in an almost miraculous result, we've managed not to annihilate each other with them so far. > Income inequality is already rising at a breathtaking pace. In the US, yes, but inequality is lessening globally. > The internet has given birth to history's most powerful surveillance system and tools of propagand…

Yeah but we were never forced into this global boiler room where we're constantly confronted with each other's thoughts and opinions. Thank you social media. It's like there is no intellectual breathing room anymore. Make anyone go mad and want to push the button..

Re: Superintelligence: The Idea That Eats Smart People

#192

Post by tim333 » Fri, Dec 23, 2016, 1:17 AM UTC

seiferteric wrote: post #147

stcredzero wrote: post #32

In 1945, as American physicists were preparing to test the atomic bomb, it occurred to someone to ask if such a test could set the atmosphere on fire...Los Alamos physicists performed the analysis and decided there was a satisfactory margin of safety. Exactly how big was this margin? Please someone tell me this was quite a large margin!

I was really surprised when I read the quote, they were worried about a runaway fusion reaction! I had heard this story before but always thought they were worried about "burning" nitrogen in the chemical sense, ie. creating nitrogen oxides. It seems doubly ridiculous to worry about fusion when we can't even get a fusion reactor to work.

We can get fusion to work in H-bombs so it might not be that silly.

Re: Superintelligence: The Idea That Eats Smart People

#193

Post by idlewords » Fri, Dec 23, 2016, 1:21 AM UTC

marcosdumay wrote: post #127

idlewords wrote: post #50

Earlier quoted context omitted.

If I remember right, something like a safety factor of 30. So not really as big as people would like! More detail in this great summary: http://large.stanford.edu/courses/2015/ph241/chung1/ And the Los Alamos paper is really good reading: http://www.sciencemadness.org/lanl1_a/lib-www/la-pubs/003290...

The summary talks about a safety factor of 1000 on an extremely unreasonable scenario of 1MeV of average temperature, falling down to 10 in an incredibly more unreasonable scenario of 10MeV temperature. I don't know the actual probabilities, but I do really expect a few atoms to get over 1MeV inside a fission bomb (as the reaction emits ~5MeV of energy). But they will almost certainly not hit each other. I also don't…

I'm trying to find where I remember the factor of 30 figure from. Maybe it was the equivalent calculation for hydrogen fusion in the ocean? I'll keep looking.

Re: Superintelligence: The Idea That Eats Smart People

#194

Post by j2kun » Fri, Dec 23, 2016, 1:21 AM UTC

AnimalMuppet wrote: post #108

narrator wrote: post #22

"The second premise is that the brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components, just like we can simulate very basic neural anatomy today." We could have a computer program that perfectly simulates the brain, but has some nasty O(2^N) complexity a…

I don't buy the AI quasi-religious stuff. But your argument here is flawed. If protein folding can do the process in constant time, we may be able to find another process (but electronic rather than wet chem) that can also do it in constant time.

I think the GP's point is that just because something is possible in principle, complexity is still a barrier in practice, probably a much more serious barrier than accepting an idea on principle

Re: Superintelligence: The Idea That Eats Smart People

#195

Post by [deleted] » Fri, Dec 23, 2016, 1:21 AM UTC

AnimalMuppet wrote: post #108

narrator wrote: post #22

"The second premise is that the brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components, just like we can simulate very basic neural anatomy today." We could have a computer program that perfectly simulates the brain, but has some nasty O(2^N) complexity a…

I don't buy the AI quasi-religious stuff. But your argument here is flawed. If protein folding can do the process in constant time, we may be able to find another process (but electronic rather than wet chem) that can also do it in constant time.

[deleted]

Re: Superintelligence: The Idea That Eats Smart People

#196

Post by idlewords » Fri, Dec 23, 2016, 1:22 AM UTC

seiferteric wrote: post #147

stcredzero wrote: post #32

In 1945, as American physicists were preparing to test the atomic bomb, it occurred to someone to ask if such a test could set the atmosphere on fire...Los Alamos physicists performed the analysis and decided there was a satisfactory margin of safety. Exactly how big was this margin? Please someone tell me this was quite a large margin!

I was really surprised when I read the quote, they were worried about a runaway fusion reaction! I had heard this story before but always thought they were worried about "burning" nitrogen in the chemical sense, ie. creating nitrogen oxides. It seems doubly ridiculous to worry about fusion when we can't even get a fusion reactor to work.

It wasn't a ridiculous thing to worry about at all.

Re: Superintelligence: The Idea That Eats Smart People

#197

Post by bhouston » Fri, Dec 23, 2016, 1:22 AM UTC

modeless wrote: post #74

alexbecker wrote: post #51

While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you grant the AI risk crowd's assumptions. Elided from https://alexcbecker.net/blog.html#against-ai-risk : The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that…

Exactly. The "Terminator" scenario of a rogue malfunctioning AI is a silly distraction from the real AI threat, which is military AIs that don't malfunction. They will give their human masters practically unlimited power over everyone else. And AI is not the only technology with the potential to worsen inequality in the world.

We don't need ai for massive differences in military effectiveness. That is already here. The us can just destroy most country and substate actors with minimal causalities. The issue is already just difficult matters like differentiating friendlies/neutrals from enemies and not creating more enemies via collateral damage and other forms of reactions.

Re: Superintelligence: The Idea That Eats Smart People

#198

Post by flukus » Fri, Dec 23, 2016, 1:23 AM UTC

coldtea wrote: post #49

tlb wrote: post #38

Earlier quoted context omitted.

Certainly the assumption that every intelligent agent will want to recursively self-improve is unwarranted. But it only takes one intelligent agent that wants to self-improve for the scary thing to happen.

> But it only takes one intelligent agent that wants to self-improve for the scary thing to happen. Only if all sorts of other conditions (several of which are mentioned in the post) also apply. Merely "wanting to self-improve" is not enough.

Wouldn't a constant desire to self improve mean a constant desire for more energy? That would bring it into conflict with other beings that want that energy.

Re: Superintelligence: The Idea That Eats Smart People

#199

Post by seiferteric » Fri, Dec 23, 2016, 1:25 AM UTC

tim333 wrote: post #192

seiferteric wrote: post #147

Earlier quoted context omitted.

I was really surprised when I read the quote, they were worried about a runaway fusion reaction! I had heard this story before but always thought they were worried about "burning" nitrogen in the chemical sense, ie. creating nitrogen oxides. It seems doubly ridiculous to worry about fusion when we can't even get a fusion reactor to work.

We can get fusion to work in H-bombs so it might not be that silly.

True, but H-bombs don't really produce runaway chain fusion reactions afaik. The conditions exist only momentarily due to the heat and pressure from the fission devices, then quickly subside and the fusion stops.

Re: Superintelligence: The Idea That Eats Smart People

#200

Post by arieskg » Fri, Dec 23, 2016, 1:26 AM UTC

Say that the simulation time period is 1 billion years. To the post-singularity, a few thousand years is the minor details in history, like what Napolean had for breakfast the morning before the Battle of Waterloo.

Let the thoughts flow as far as you'd like, but come back to reality once in a while. After all, your ideas are conditional to your body.