Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People

idlewords.com

Post reply on HN
Print view View original

541–550 of 607 posts

Re: Superintelligence: The Idea That Eats Smart People

#541

Post by SerLava »

lmm wrote: post #482
SerLava wrote: post #60

Earlier quoted context omitted.

I don't understand why people have such a weird problem reconciling the brain with the mind. It IS all physical. It's also an unimaginably complex fucking shitload of tiny physical objects working incredibly quickly. If your brain was big enough to see the parts working like little gears of a clock, it would probably be planet sized, or something like that. Huge. I would EVEN say it doesn't raise any interesting phil…

> Semi-related, if you made a giant brainlike calculator out of water pipes and valves and shit, and asked it a question, then turned all its pipes and valves for a while... it would probably just say "please kill me." Um why? Most humans don't do that.

Well it would kind of be a brain trapped in sensory deprived hell.

The point is I think people would expect pretty mundane things from a giant "mechanical" brain, but as described, it would be very unpredictable and complex. It would do things that instantly make the experimenters uneasy.

Re: Superintelligence: The Idea That Eats Smart People

#542

Post by witty_username »

Ygg2 wrote: post #372
witty_username wrote: post #332

Earlier quoted context omitted.

Global income inequality has been decreasing. http://voxeu.org/article/parametric-estimations-world-distri...

That's inequality between states. Inequality within states hasn't decreased.

IIRC inequality of all humans has decreased as poor people have become richer.

Re: Superintelligence: The Idea That Eats Smart People

#543

Post by MisterBastahrd »

The problem with military coups is that a lot of the people who are involved in them never survive the regime they create. When you are trying to consolidate power as a figurehead, the first thing you do is to make sure that anyone with questionable loyalties, whether true or not, are eliminated and replace with people who are grateful to hold that position.

Intelligent people know this, which is why there really aren't that many military coups, regardless of how those in the military feel about their political masters.

Re: Superintelligence: The Idea That Eats Smart People

#544

Post by copenja »

breuleux wrote: post #224
openasocket wrote: post #187

Earlier quoted context omitted.

> Why would the "aristocracy" owning the factories care about money, when they have all the goods (or can trade for them with other aristos)? In this scenario, they only need enough factories to meet the needs of themselves and the other aristocrats. Which means there will be massive demand for goods for the common people, which means there will be jobs for people. The only way your scenario works is if the aristocra…

> In this scenario, they only need enough factories to meet the needs of themselves and the other aristocrats. That could still end up taking all of the planet's resources, if enough aristocrats try to maximize their military capacity to protect themselves or play some kind of planet-wide game of Risk, or if they create new goods that require absolutely insane amounts of resources. I mean, ultimately, the issue is th…

What about a scenario in which the common people revolt and kill the aristocrats?

It has happened before.

Re: Superintelligence: The Idea That Eats Smart People

#545

Post by marcosdumay »

idlewords wrote: post #193
marcosdumay wrote: post #127

Earlier quoted context omitted.

The summary talks about a safety factor of 1000 on an extremely unreasonable scenario of 1MeV of average temperature, falling down to 10 in an incredibly more unreasonable scenario of 10MeV temperature. I don't know the actual probabilities, but I do really expect a few atoms to get over 1MeV inside a fission bomb (as the reaction emits ~5MeV of energy). But they will almost certainly not hit each other. I also don't…

I'm trying to find where I remember the factor of 30 figure from. Maybe it was the equivalent calculation for hydrogen fusion in the ocean? I'll keep looking.

It's well within the 1000 - 10 range. May be an overall approximation somewhere on the paper.

But I was just summarizing it. Looks like the kind of calculation one does when he's entirely convinced it's impossible, but needs to verify anyway. I liked reading it, and I was surprised they didn't take meteors into account.

Re: Superintelligence: The Idea That Eats Smart People

#546

Post by darawk »

Retra wrote: post #506
darawk wrote: post #125

Earlier quoted context omitted.

Sure, but none of those laws would prevent a superintelligence from generating more than enough electricity to power itself, in principle.

AI's powering themselves isn't the objection. Exponential increase in capability is. And that requires you to deal with things like heat dissipation, which are real problems for our current "dumb" computers.

It's not a huge problem for humans. I see no reason to think that a computer say, 10x as powerful as a human would have major problems with heat dissipation.

Re: Superintelligence: The Idea That Eats Smart People

#547

Post by eli_gottlieb »

otakucode wrote: post #405

He is completely skipping the most important part. The superintelligence has to have some reason to be in conflict with us. Human beings don't go out of their way to hunt down and eliminate ants. They don't find out what ants eat and sieze control of it to manipulate them. There is no reason to think that a superintelligent machine would be likely to present terrible interference to us proposed. So it's super smart a…

>He is completely skipping the most important part. The superintelligence has to have some reason to be in conflict with us. Human beings don't go out of their way to hunt down and eliminate ants. They don't find out what ants eat and sieze control of it to manipulate them. There is no reason to think that a superintelligent machine would be likely to present terrible interference to us proposed.

That depends on whether eliminating us is simple, like stepping on an ant-hill you didn't realize was there, or complicated, like going to war.

I don't buy the argument that for an AI to eliminate us would be relatively simple, that it would be as subjectively simple as we adults find reading and writing, while still involving cognitive effort. That would still require deliberate effort.

The trouble is whether the AI wants, for instance, our electricity infrastructure, and doesn't care about the riots and social breakdown caused when we humans can't get electricity for our own needs anymore. It didn't go into conflict with us. It just killed a lot of people (for instance, everyone on life-support in a hospital reliant on the electrical grid) without noticing or caring.

Likewise, maybe it decides that rising sea-levels and accelerating global warming are good for its own interests somehow. That stuff will kill us, but it doesn't require a complicated strategy, it just requires setting lots of things on fire. Any moron can do that, but most human morons don't want to.

I certainly agree that a "hostile" AI which perceives itself as requiring complicated, deliberate effort to kill all humans will probably not kill all humans. It'll find some lazier, simpler way to get what it wants.

Re: Superintelligence: The Idea That Eats Smart People

#548

Post by mundo »

kobayashi wrote: post #279
mundo wrote: post #151

Earlier quoted context omitted.

People will stop downvoting this if you edit it to add a specific example or three.

Too late to edit, so I'll post just a few examples here: >The only way out of this mess is to design a moral fixed point, so that even through thousands and thousands of cycles of self-improvement the AI's value system remains stable, and its values are things like 'help people', 'don't kill anybody', 'listen to what people want'. Bostrom absolutely did not say that the only way to inhibit a cataclysmic future for hu…

> I find it completely unbelievable that anyone who read Superintelligence could possibly assert "The Argument From My Roommate" with a straight face

Pretty sure that was a joke, and zeroing in on it is a pretty bad violation of the principle of charity. A lot of the other items in the talk (e.g. "like the alchemists, we don't even understand this well enough to have realistic goals" and "counting up all future human lives to justify your budget is a bullshit tactic" and "there's no reason to think an AI that qualifies as superintelligent by some metric will have those sorts of motives anymore") seem to me to be fair and rather important critiques of Bostrom's book. (although I was admittedly already a skeptic on this)

Re: Superintelligence: The Idea That Eats Smart People

#549

Post by mahyarm »

driverdan wrote: post #522
mahyarm wrote: post #257

Earlier quoted context omitted.

Calfornia is mismanged to hell. The bay area has some of the worst roads in the nation with very mild weather and a wealthy tax base. It cost $8billion to make 1 or 2 miles of central subway and only €11 billion to make the worlds longest tunnel under the alps. I have the same income tax rate as I did in canada yet there isnt universal healthcare and far more economic inequality. It goes on and on. If you trippled th…

> The bay area has some of the worst roads in the nation You haven't traveled much if you believe this. Try going to the northeast sometime.

SF/Oakland tops the list in worst roads in metros over 500k pop:

http://www.businessinsider.com/the-worst-roads-in-america-20...

That there are bad roads elsewhere? I believe that too :).

The thing is it shouldn't be topping any list like this, with the amount of money and lack of north eastern weather.

Re: Superintelligence: The Idea That Eats Smart People

#550

Post by sctb »

erikbye wrote: post #415
singlow wrote: post #131

Earlier quoted context omitted.

Their might is asymmetrical, but power is mitigated by the willingness of an organization of humans to follow commands. There is a limit to how far a soldier will go, ethically.

Which limit are you talking about? I guess you have forgotten what the soldiers of the third reich did. Or American death squads in Afghanistan. Might as well call them rape and death squads. What about Guantanamo? Srebrenica? Should I go on? How unenlightened and naive are you?

Please comment civilly and substantively—without personal attacks—or not at all.

https://news.ycombinator.com/newswelcome.html

https://news.ycombinator.com/newsguidelines.html

Post reply on HN