Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People

idlewords.com

Post reply on HN
Print view View original

171–180 of 607 posts

Re: Superintelligence: The Idea That Eats Smart People

#171

Post by Almaviva »

seiferteric wrote: post #65
md224 wrote: post #27

"Put another way, this is the premise that the mind arises out of ordinary physics... If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise to accept." The thing that irks me about this is how it reinforces a common (and in my opinion, false) dichotomy: either you believe the mind is explicable in terms of ordinary physics or you believe…

One thing I rarely see discussed is the possibility that the brain is just too complex to practically reproduce. That is to say, that it is technically possible, but not practical. Evolution had billions of years, working in a massively parallel way to work this out after all. It's possible that the brain is a huge tangled mess of rules and special cases that we will never be able to fully understand and reproduce. A…

> Evolution had billions of years, working in a massively parallel way to work this out after all.

And it also got to work on it with a hundred billion stars with earth-like planets, in each of a hundred billion galaxies, and (maybe) in a basically infinite space of universes. All in parallel with infinite time to play with.

Is it so far-fetched to postulate that humans will hit our limits before getting this far?

Re: Superintelligence: The Idea That Eats Smart People

#172

Post by imh »

apsec112 wrote: post #53

"I live in California, which has the highest poverty rate in the United States, even though it's home to Silicon Valley. I see my rich industry doing nothing to improve the lives of everyday people and indigent people around us." This is trivially false. Over a hundred billionaires have now pledged to donate the majority of their wealth, and the list includes many tech people like Bill Gates, Larry Ellison, Mark Zuck…

You're talking about the people and he's talking about the industry. There's a difference between Microsoft and Bill Gates.

Re: Superintelligence: The Idea That Eats Smart People

#173

Post by nradov »

singlow wrote: post #131
zevyoura wrote: post #123

Earlier quoted context omitted.

Don't the people who control military technology in our current time already have "practically unlimited power over everyone else"?

Their might is asymmetrical, but power is mitigated by the willingness of an organization of humans to follow commands. There is a limit to how far a soldier will go, ethically.

Really?

https://en.wikipedia.org/wiki/Nanking_Massacre

Re: Superintelligence: The Idea That Eats Smart People

#174

Post by paganel »

gbhn wrote: post #145
singlow wrote: post #131

Earlier quoted context omitted.

Their might is asymmetrical, but power is mitigated by the willingness of an organization of humans to follow commands. There is a limit to how far a soldier will go, ethically.

True, true, but somehow that's never been much of an obstacle to totalitarian governments. Somehow there's always a soldier who will push the button. In Nuremberg we developed a way to think about this: people outside the central circles of power have a tremendous amount of pressures they are considering. It's definitely the case that some are sadistic and horrible, but more are just following orders and trying to ge…

> Somehow there's always a soldier who will push the button.

And then there's always that soldier ready to denounce his camarads for having raped some poor Vietnamese women who had nothing to do with the war itself. Why risk such a PR disaster which might see your funding cut when you can use robots instead? Warzone robots don't snitch on their fellow robots in front of the press and they don't rape, they're only build to kill.

Re: Superintelligence: The Idea That Eats Smart People

#175

Post by _cbsz »

paulbaumgart wrote: post #161
mrfusion wrote: post #91

Earlier quoted context omitted.

What's his reason? I got halfway through that and realized I didn't understand.

That even if you see diminishing returns to what you can accomplish with greater intelligence, a small advantage will still compound into a large one over the theoretically infinite lifespan of an AI.

> the theoretically infinite lifespan of an AI

Infinite? Why isn't AI subject to the heat death of the universe?

Re: Superintelligence: The Idea That Eats Smart People

#176

Post by monk_e_boy »

Afforess wrote: post #159

> Observe that in these scenarios the AIs are evil by default, just like a plant on an alien planet would probably be poisonous by default. I believe this is a core misunderstanding. Bostrom never says that a superintelligent AI is evil by default. Bostrom argues the AI will be orthogonal, it's goals will be underspecified in such a way that leads it to destroy humanity. The paperclip optimizer AI doesn't want to kil…

An uncaring blind AI wouldn't be very interesting, I would assume whoever had turned it on and was observing it would just turn it off and tweek the algorithm.

But I don't think this is even possible as when the AI was in child stage and was learning, it would learn from people, so it would become like people. Or at least understand them. At some point it has to know less than a human and will learn at a rate that humans can measure. As we measure we can make decisions about it.

I don't agree with the assumption that a factory making paperclips will make the transition to a super intelligent AI in a short time frame. I think it will take years. And along that route (of years of learning) we'll have time to talk to it and decide if it should be kept switched on.

Re: Superintelligence: The Idea That Eats Smart People

#177

Post by modeless »

zardo wrote: post #156
singlow wrote: post #131

Earlier quoted context omitted.

Their might is asymmetrical, but power is mitigated by the willingness of an organization of humans to follow commands. There is a limit to how far a soldier will go, ethically.

>There is a limit to how far a soldier will go, ethically. What horrible things to you have in mind that armies have not already done?

It's not that AIs will do worse things than humans have already done. It's that AIs will do those terrible things much more efficiently and with much lower risk to the people in charge.

Re: Superintelligence: The Idea That Eats Smart People

#178

Post by Analemma_ »

k8t wrote: post #140
idlewords wrote: post #95

Earlier quoted context omitted.

The problem with this precautionary reasoning is it leads you to Pascal's Mugging, where you are ready to believe very unlikely things because of the enormous impact they'll have if true.

Yes, I agree with the reasoning behind Pascal's Mugging. But Pascal's Mugging refers to things that have an astronomically low chance of happening — like 0.000000001. But is the chance that AI is dangerous that low? Nobody in the world knows for sure at this point, due to how far away superAI might be, and due to the uncertainties in implementation. Therefore, if we use Bayesian thinking and spread it out, I'm not re…

There's no magic probability value at which the Pascal's Mugging argument suddenly "kicks in". It's all about the utilitarian tradeoff of, "given this low probability, am I devoting the right amount of resources to preventing this terrible event? Could they be better spent elsewhere? And is the fretting harmful in and of itself, wiping out the expected gain?" The talk is arguing that the answer is yes to both of those latter questions.

Re: Superintelligence: The Idea That Eats Smart People

#179

Post by ph0rque »

alexbecker wrote: post #51

While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you grant the AI risk crowd's assumptions. Elided from https://alexcbecker.net/blog.html#against-ai-risk : The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that…

>We already have nuclear weapons, which like almost everything else are always getting cheaper to produce.

And in an almost miraculous result, we've managed not to annihilate each other with them so far.

> Income inequality is already rising at a breathtaking pace.

In the US, yes, but inequality is lessening globally.

> The internet has given birth to history's most powerful surveillance system and tools of propaganda.

It has also given birth to a lot of good things, some that are mentioned in a sibling comment.

Re: Superintelligence: The Idea That Eats Smart People

#180

Post by citrin_ru »

panic wrote: post #124
alexbecker wrote: post #51

While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you grant the AI risk crowd's assumptions. Elided from https://alexcbecker.net/blog.html#against-ai-risk : The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that…

It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that its masters start thinking about the next group, and the next. This has been done many times by human-run militaries; would AI make it worse somehow? Groups of humans acting collectively can look a lot like an "AI" from the right perspective. Corporations focused on optimizing their profit spend a huge amo…

> This has been done many times by human-run militaries; would AI make it worse somehow?

Solders in developed countries no longer want to die en masse.

Post reply on HN