Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superintelligence: The Idea That Eats Smart People

idlewords.com

Post reply on HN
Print view View original

231–240 of 607 posts

Re: Superintelligence: The Idea That Eats Smart People

#231

Post by SilasX »

trishume wrote: post #215
tlb wrote: post #38

Earlier quoted context omitted.

Certainly the assumption that every intelligent agent will want to recursively self-improve is unwarranted. But it only takes one intelligent agent that wants to self-improve for the scary thing to happen.

Why wouldn't it if it is able to? It doesn't have to "want" to self-improve, it only has to want anything that it could do better if it was smarter. All it needs is the ability, the lack of an overwhelming reason not to, and a basic architecture of optimizing towards a goal. If you knew an asteroid would hit the earth 1 year from now, and you had the ability to push a button and become 100,000x smarter, I would hope…

Because there are tradeoffs. Whatever it's goal is, some of those "drives" (instrumental values) will be more effective for reaching that goal over the timespan that it cares about.

Sometimes "accumulating more resources" is the most effective way. Sometimes "better understanding what problem I'm trying to solve" is the most effective way". Sometimes "resisting attempts to subvert my mind" is the most effective way. And yes, sometimes "becoming better at general problem solving" (self improvement of one's intelligence) is the most effective way.

But there's no guarantee that any one of those will be a relevant bottleneck in any particular domain, so there's no guarantee an agent will pursue all of those drives.

Re: Superintelligence: The Idea That Eats Smart People

#235

Post by naasking »

modeless wrote: post #74
alexbecker wrote: post #51

While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you grant the AI risk crowd's assumptions. Elided from https://alexcbecker.net/blog.html#against-ai-risk : The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that…

Exactly. The "Terminator" scenario of a rogue malfunctioning AI is a silly distraction from the real AI threat, which is military AIs that don't malfunction. They will give their human masters practically unlimited power over everyone else. And AI is not the only technology with the potential to worsen inequality in the world.

> The "Terminator" scenario of a rogue malfunctioning AI is a silly distraction from the real AI threat, which is military AIs that don't malfunction. They will give their human masters practically unlimited power over everyone else.

To be fair, it's a small step from effective AI that doesn't malfunction, to an AI over which humans have lost control. It's precisely one vaguely specified command away in fact, and humans are quite excellent at being imprecise.

Re: Superintelligence: The Idea That Eats Smart People

#236

Post by hueving »

pmyjavec wrote: post #77
apsec112 wrote: post #53

"I live in California, which has the highest poverty rate in the United States, even though it's home to Silicon Valley. I see my rich industry doing nothing to improve the lives of everyday people and indigent people around us." This is trivially false. Over a hundred billionaires have now pledged to donate the majority of their wealth, and the list includes many tech people like Bill Gates, Larry Ellison, Mark Zuck…

I feel the problem is the fact there is 100 billionaires in he first place, no one gets rich on their own. Gates et al, are clever, but didn't get where they are totally independently without others support, so they should give back. Also, some of these billionaires are running companies that are great at tax avoidance, probably most of them. Now what? They get to pick and choose where they get to spend there/invest…

Can you explain to me why you think taking all of the wealth of 100 billionaires will help the poor? 100 billion spread over the population of California is less than $3000/person. So you can wipe out all of the billionaires and give everyone $50/week for one year, what is that going to change?

Re: Superintelligence: The Idea That Eats Smart People

#238

Post by kenko »

alexbecker wrote: post #51

While I agree with Maciej's central point, I think the inside arguments he presents are pretty weak. I think that AI risk is not a pressing concern even if you grant the AI risk crowd's assumptions. Elided from https://alexcbecker.net/blog.html#against-ai-risk : The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that…

> It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that its masters start thinking about the next group, and the next

You know, you don't need to go that far. You know what a great way to kill a particular group of people is? Well, let's take a look at what a group of human military officers decided to do (quoting from a paper of Elizabeth Anscombe's, discussing various logics of action and deliberation):

""" Kenny's system allows many natural moves, but does not allow the inference from "Kill everyone!" to "Kill Jones!". It has been blamed for having an inference from "Kill Jones!" to "Kill everyone!" but this is not so absurd as it may seem. It may be decided to kill everyone in a certain place in order to get the particular people that one one wants. The British, for example, wanted to destroy some German soldiers on a Dutch island in the Second World War, and chose to accomplish this by bombing the dykes and drowning everybody. (The Dutch were their allies.) """

There's a footnote:

""" Alf Ross shews some innocence when he dismisses Kenny’s idea: ‘From plan B (to prevent overpopulation) we may infer plan A (to kill half the population) but the inference is hardly of any practical interest.’ We hope it may not be. """

It's not an ineffective plan.

Re: Superintelligence: The Idea That Eats Smart People

#239

Post by trishume »

SilasX wrote: post #231
trishume wrote: post #215

Earlier quoted context omitted.

Why wouldn't it if it is able to? It doesn't have to "want" to self-improve, it only has to want anything that it could do better if it was smarter. All it needs is the ability, the lack of an overwhelming reason not to, and a basic architecture of optimizing towards a goal. If you knew an asteroid would hit the earth 1 year from now, and you had the ability to push a button and become 100,000x smarter, I would hope…

Because there are tradeoffs. Whatever it's goal is, some of those "drives" (instrumental values) will be more effective for reaching that goal over the timespan that it cares about. Sometimes "accumulating more resources" is the most effective way. Sometimes "better understanding what problem I'm trying to solve" is the most effective way". Sometimes "resisting attempts to subvert my mind" is the most effective way.…

Agreed. But if the goal is something like "build the largest number of paperclips", recursive self improvement is going to be a phenomenally good way to achieve that, unless it is already intelligent enough to be able to tile the universe with paperclips. Either way we don't care if it self improved or not, that's just the seemingly most likely path, we just care if it is overwhelmingly more powerful than us.

The only thing that stops me from recursively self improving is that I'm not able to. If I could it would be a fantastic way to do good things that as an altruistic human I want to do. Like averting crises (climate change, nuclear war), minimizing poverty and misery, etc...

Re: Superintelligence: The Idea That Eats Smart People

#240

Post by EGreg »

bhouston wrote: post #197
modeless wrote: post #74

Earlier quoted context omitted.

Exactly. The "Terminator" scenario of a rogue malfunctioning AI is a silly distraction from the real AI threat, which is military AIs that don't malfunction. They will give their human masters practically unlimited power over everyone else. And AI is not the only technology with the potential to worsen inequality in the world.

We don't need ai for massive differences in military effectiveness. That is already here. The us can just destroy most country and substate actors with minimal causalities. The issue is already just difficult matters like differentiating friendlies/neutrals from enemies and not creating more enemies via collateral damage and other forms of reactions.

The problem arises when non-state actors wreak havoc with drones and AI with inpunity.

What would we do if a drone made by a no-name manyfacturer dropped some bombs in Times Square? Who would we blame when someone uses AI to actually sow social mistrust and subvert our existing systems?

Post reply on HN