Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rationality: From AI to Zombies

intelligence.org

Post reply on HN
Print view View original

121–130 of 142 posts

Re: Rationality: From AI to Zombies

#121

Post by jtheory »

zyxley wrote: post #117
Strilanc wrote: post #107

Earlier quoted context omitted.

I'm not trying to put forth a framework, I'm trying to explain the side you're not on without straw-manning them. - If you prefer 100000000000000000000000000000000000000 specks to 1 torture - and you follow the axiom that A > B implies x% A > x% B, and value people equally - then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture - and you will want to throw a speck in someone's eyes…

What I see here is proof that this is a bad model that produces insane results, rather than it proving that I've made a bad decision.

The whole point is that the model is bad.

The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye".

Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize:

* some people spontaneously torture others

* the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them

* so: eyeball dust storms commencing in 3, 2, 1...

One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.

Re: Rationality: From AI to Zombies

#122

Post by maxerickson »

jtheory wrote: post #120
maxerickson wrote: post #29

Earlier quoted context omitted.

If you take seriously the idea of an infohazard, you have a problem, because the idea that information can be hazardous is itself hazardous information (at a minimum, it's a prompt for us sloppy thinkers to start imagining what sort of information might be hazardous). Something as innocuous as "Think safe thoughts!" enjoys a similar property.

That doesn't make sense, though -- info hazards are not normally things you might just think of, or we'd be screwed either way; "whatever you do, don't think of a pink elephant", right? Toss out silly ideas like the basilisk thing, and think up real info hazard possibilities. E.g., let's say you knew what Snowden was up to, a few weeks before he figured out how to publish what he'd found and leave the US. You'd have…

Yeah, it would have been better formulated using idea hazard instead of info hazard.

Re: Rationality: From AI to Zombies

#123

Post by jtheory »

zyxley wrote: post #91
TeMPOraL wrote: post #56

Earlier quoted context omitted.

One reason is that you'd need to prove that the lastic impact is in fact zero. 3^^^3 dust specks is enough for one of them to rub something in a wrong way that rubs something else in the wrong way that gives a person cancer.

Having a speck of dust in one's eye will change the course of life in a tiny way for any given person, true, but unless we're postulating an even more absurd edge case like "everyone blinking at exactly the same time makes a blip in the physics experiment machine go unnoticed and a week later the universe explodes", it's a change that's going to be effectively random and indistinguishable from the general minor messi…

You're not making a serious effort to guess at what difference it might make, though.

Example (this is actually true!) -- I have a life-long inflammation in both eyes that started in one eye when I was about 5, and eventually spread to the other.

The specialists weren't able to figure out the original cause; their best guess was basically a speck of dust -- just the wrong speck of dust (maybe with a microorganism in it?), in the wrong place, that my immune system attacked and then just kept on attacking even after the original speck had been gone for years. Medication plus a series of unpleasant surgeries have managed to keep my eyes relatively functional, so far, but others with the same condition certainly do lose all vision.

Obviously this isn't a common result; but given enough specks of dust.... Well, you have to start doing some math.

Re: Rationality: From AI to Zombies

#124

Post by kybernetikos »

ikeboy wrote: post #108
kybernetikos wrote: post #101

Earlier quoted context omitted.

There's a difference between the utility of a world where one person has a mote of dust in their eye and the utility of a world where one person has a mote of dust in their eye because it saves someone else . >At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and t…

So your intuition says that there's some point where everyone is suffering pain X, where X is very large, and they would each agree to cause the X-(1 mote speck) to a trillion times as many people, in return for reducing their own suffering by 1 mote speck? That strikes me as beyond regular selfishness, and non intuitive.

Can you explain more clearly why what I said implies that, because I don't think I mean to say that.

I believe that there is an amount of suffering that it is reasonable to expect anyone to accept in order to help another person. That amount depends on the amount to be suffered, and the amount benefited by the recipient. Once the suffering falls under that threshold, I do not believe the number of people required to make the sacrifice comes into consideration, as each of them if reasonable would say "I prefer to belong to this world, where as part of a huge group I accept this small ill in order that someone else benefits". Therefore, the implied sacrifice results in greater utility for that choice.

Let me try a different tack.

Let's say that you observe a universe with some large number of people suffering dust specks in their eye. That sounds bad. But what if every single one of those people actually suffering thinks that this universe is better than the alternatives. You don't suffer from a dust spec, but are you going to ignore all those people in their estimation of the utility of the universe? If you switched to a universe where all of those people didn't suffer from dust specs, but someone else was suffering, they would tell you that that was a worse universe.

It's pretty obvious to me that even if that isn't the exact case, it's close to being the case in reality - that's why people find the dust speck argument to be unintuitive, not the large numbers thing. It's because some measure of sacrifice for other people is part of what we expect from everyone, and most people know instinctively that if everyone asked to make a sacrifice agrees that it's right to make that sacrifice, then the world is better because of it.

Re: Rationality: From AI to Zombies

#125

Post by kybernetikos »

ikeboy wrote: post #109
zyxley wrote: post #102

Earlier quoted context omitted.

> and that point is the point at which I determine that the hurt falls under the threshold I would expect every person to be prepared to sacrifice for any other person. That seems like a good way to put it. The number of people doesn't really matter, when it the amount of hurt per person is so small that you can say with all confidence something like "I believe that literally any sane person would agree to get one sp…

So first of all, your preferences are still circular, unless you bite another bullet somewhere. But even your argument isn't quite accurate. It's not this one person who needs to get a speck, it's a literally unimaginable amount of people. As it happens to be, a number of people in our tiny world have said they would choose torture, so your argument fails just considering them.

> But even your argument isn't quite accurate. It's not this one person who needs to get a speck, it's a literally unimaginable amount of people.

Nearly all of whom prefer to receive the speck than to allow the individual to be tortured.

> As it happens to be, a number of people in our tiny world have said they would choose torture, so your argument fails just considering them.

Yes, I feel somewhat uncomfortable ignoring the agency of torturers and murderers in this scenario, and that reluctance would play into a very conservative estimate of where that threshold should be, but I would be reluctant to choose a lowest common denominator measure to establish morality.

Re: Rationality: From AI to Zombies

#126

Post by philh »

kybernetikos wrote: post #90
pyrois wrote: post #73

Earlier quoted context omitted.

He's assuming linearity, which at the very least needs justification. He's assuming that the function that maps from the pair (number of people, type of torture) to suffering is linear in the number of people, and also linear in the type of torture. I don't believe either of those things are true. To put it more clearly, he says: > So we can keep doing this, gradually - very gradually - diminishing the degree of disc…

We're social animals. There's an amount of discomfort (maybe not large) I would be prepared to undergo to help someone else, and I find myself thinking that others "should" also be prepared to undergo such an amount of discomfort to assist others. Given the choice to accept a dust mote temporarily in my eye as part of a huge crowd in order to save another person from torture, I would gladly accept that, and I think a…

> I would gladly accept that, and I think all reasonable people would too

Hello, apparently I'm unreasonable. And so is everyone else who has said that they choose torture over dust specks. If you select 3^^^3 people, you're going to find an awful lot of us. (And also some sociopaths who literally don't care if someone else gets tortured.)

Your argument seems to boil down to "specks is the correct answer, so anyone who gets it wrong doesn't count; and because we all agree that specks is the correct answer, it's okay to do specks".

On the other hand, I would totally accept the torture for myself, if it would prevent the specks. (At least I hope I would, and to the extent that I can model how I would act in that situation, it does seem plausible that I would.)

Re: Rationality: From AI to Zombies

#127

Post by convexfunction »

zyxley wrote: post #105
convexfunction wrote: post #43

Earlier quoted context omitted.

Eliezer agrees with you! Roko's Basilisk in particular is wrong. As a matter of policy though, you do not just publicly publish information hazards that might be real ; that was why it was a big deal.

> you do not just publicly publish information hazards that might be real But... for "might be real" to be valid at all, you have to buy into the Roko's basilisk concept in the first place. If you don't, there's nothing to get upset about.

I meant "might be real" according to generic-you in that statement. Someone being wrong doesn't make information hazards in general disappear.

Re: Rationality: From AI to Zombies

#128

Post by TheOtherHobbes »

Neslit wrote: post #84
TheOtherHobbes wrote: post #77

Earlier quoted context omitted.

Actually my decision probably depends on the person. [cough] But anyway. This isn't even philosophy - it's a digital remix of medieval scholasticism pretending to be philosophy. The irony is that politics proves empirically that ideas actually can be dangerous and harmful. And some ideas - actually narratives - can be very dangerous and harmful indeed. There's over a century of "persuasion technology" (Bernays, etc)…

I notice you didn't actually say at which point you would prefer torturing (10^100)*X people for Y years each, over torturing X people for Y+0.0000001 years each, for X and Y at least 0.0000001. (You may assume you don't know anything in particular about these people, other than that they are adult humans.)

Of course I didn't. When dealing with real moral issues the question is wholly trivial.

The fact that it includes some numbers that reduce to some other numbers doesn't change that.

Putting numbers into something doesn't make it scientific or objective. It just makes it numerical.

There is a difference, and it's not a small one.

Re: Rationality: From AI to Zombies

#129

Post by zyxley »

jtheory wrote: post #121
zyxley wrote: post #117

Earlier quoted context omitted.

What I see here is proof that this is a bad model that produces insane results, rather than it proving that I've made a bad decision.

The whole point is that the model is bad. The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye". Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize: * some people spontaneously torture others * the chances of spontaneous torture is a bit lower if all people a…

> The whole point is that the model is bad.

That doesn't seem to be Yudkowsky's argument.

Re: Rationality: From AI to Zombies

#130

Post by Neslit »

TheOtherHobbes wrote: post #128
Neslit wrote: post #84

Earlier quoted context omitted.

I notice you didn't actually say at which point you would prefer torturing (10^100)*X people for Y years each, over torturing X people for Y+0.0000001 years each, for X and Y at least 0.0000001. (You may assume you don't know anything in particular about these people, other than that they are adult humans.)

Of course I didn't. When dealing with real moral issues the question is wholly trivial. The fact that it includes some numbers that reduce to some other numbers doesn't change that. Putting numbers into something doesn't make it scientific or objective. It just makes it numerical. There is a difference, and it's not a small one.

What the heck? Sorry, I didn't understand any of this. Are you saying that with real moral issues, it always trivially wrong to torture? This seems simply false.

If I capture person X and X's laptop Y, and X tells me under no duress that Y contains the location of a nuclear bomb that X has placed in a major city, and I have other strong evidence that this is true, but X refuses to give me the password to Y; then it is moral (but rightly illegal) for me to torture X for the password to Y.

Torture is not literally incommensurate with any other bad thing. Then the question arises, how do we, in full generality, determine which is the greater of two evils (or the better of two goods)? The torture vs. dust specks thing is supposed to disabuse people of the unhelpful notion that some things are somehow incomparable in terms of goodness and badness.

Post reply on HN