Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rationality: From AI to Zombies

intelligence.org

Post reply on HN
Print view View original

91–100 of 142 posts

Re: Rationality: From AI to Zombies

#91

Post by zyxley »

TeMPOraL wrote: post #56
zyxley wrote: post #41

Earlier quoted context omitted.

> Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero Why not? I'm entirely comfortable saying that the lasting lifetime impact of a single speck of dust in someone's eye is zero, given that it has no lasting physiological consequences and in short order will be forgotten about completely.

One reason is that you'd need to prove that the lastic impact is in fact zero. 3^^^3 dust specks is enough for one of them to rub something in a wrong way that rubs something else in the wrong way that gives a person cancer.

Having a speck of dust in one's eye will change the course of life in a tiny way for any given person, true, but unless we're postulating an even more absurd edge case like "everyone blinking at exactly the same time makes a blip in the physics experiment machine go unnoticed and a week later the universe explodes", it's a change that's going to be effectively random and indistinguishable from the general minor messiness of human life.

Re: Rationality: From AI to Zombies

#92

Post by ikeboy »

kybernetikos wrote: post #90
pyrois wrote: post #73

Earlier quoted context omitted.

He's assuming linearity, which at the very least needs justification. He's assuming that the function that maps from the pair (number of people, type of torture) to suffering is linear in the number of people, and also linear in the type of torture. I don't believe either of those things are true. To put it more clearly, he says: > So we can keep doing this, gradually - very gradually - diminishing the degree of disc…

We're social animals. There's an amount of discomfort (maybe not large) I would be prepared to undergo to help someone else, and I find myself thinking that others "should" also be prepared to undergo such an amount of discomfort to assist others. Given the choice to accept a dust mote temporarily in my eye as part of a huge crowd in order to save another person from torture, I would gladly accept that, and I think a…

You aren't quite grokking the difference between "huge crowd" and 3||3. How do you deal with the argument from circularity? Are your preferences circular, and if not, which part do you reject?

Re: Rationality: From AI to Zombies

#93

Post by unbudgingprawn »

kenbellows wrote: post #71
unbudgingprawn wrote: post #23

Earlier quoted context omitted.

It seems like he deeply believes in it. I don't think he realizes it's just a modern version of Pascal's wager

In attempting to debunk this sort of statement, he has literally written the following: "In the hands of RationalWiki generally, and RationalWiki leader David Gerard particularly ..., [the fictional version of the Roko's Basilisk event] somehow metamorphosed into a Singularity cult that tried to get people to believe a Pascal's Wager argument to donate to their AI god on pain of torture. This cult that has literally…

"This sentence is a lie." He doesn't seem to actually debunk the idea, just yells about David Gerard for a bit. Can you explain how it's different from Pascal's Wager?

Re: Rationality: From AI to Zombies

#94

Post by ikeboy »

pyrois wrote: post #73
ikeboy wrote: post #60

Earlier quoted context omitted.

What do you say to the argument in http://lesswrong.com/lw/n3/circular_altruism/ : >But let me ask you this. Suppose you had to choose between one person being tortured for 50 years, and a googol people being tortured for 49 years, 364 days, 23 hours, 59 minutes and 59 seconds. You would choose one person being tortured for 50 years, I do presume; otherwise I give up on you. >And similarly, if you had to choose betwe…

He's assuming linearity, which at the very least needs justification. He's assuming that the function that maps from the pair (number of people, type of torture) to suffering is linear in the number of people, and also linear in the type of torture. I don't believe either of those things are true. To put it more clearly, he says: > So we can keep doing this, gradually - very gradually - diminishing the degree of disc…

It doesn't need to be linear, just not slowing super-exponentially. At almost any slowdown rate, specks will still end up worse than torture.

Re: Rationality: From AI to Zombies

#95

Post by kybernetikos »

ikeboy wrote: post #92
kybernetikos wrote: post #90

Earlier quoted context omitted.

We're social animals. There's an amount of discomfort (maybe not large) I would be prepared to undergo to help someone else, and I find myself thinking that others "should" also be prepared to undergo such an amount of discomfort to assist others. Given the choice to accept a dust mote temporarily in my eye as part of a huge crowd in order to save another person from torture, I would gladly accept that, and I think a…

You aren't quite grokking the difference between "huge crowd" and 3||3. How do you deal with the argument from circularity? Are your preferences circular, and if not, which part do you reject?

The size of the numbers is irrelevant. I personally would consider a world in which I had no mote of dust and someone else was tortured as a world with less utility than a world in which I had a mote of dust in my eye. I expect every single one of those 3||3 individuals to feel the same way. Therefore, there is no number of individuals that I would choose not to suffer the dust mote in preference to torturing someone.

The answer to the argument from circularity is obvious - there is a discontinuity. I can see the discontinuity in my own thinking, and you probably can too. I would accept a dust mote to save an individual from years of torture (and I think almost all reasonable people would), but I would not accept a dust mote to save two people from a dust mote. That may be unethical of me, since I would of course prefer the world that has fewer people with dust motes, but for me to expect the greater number of people to make the sacrifice, I must be prepared to make the sacrifice myself and it must be a sacrifice that I think all people should make. Exactly how much I think people should sacrifice for others is difficult to say, but there is a clear step change at some point.

Re: Rationality: From AI to Zombies

#96

Post by ikeboy »

kybernetikos wrote: post #95
ikeboy wrote: post #92

Earlier quoted context omitted.

You aren't quite grokking the difference between "huge crowd" and 3||3. How do you deal with the argument from circularity? Are your preferences circular, and if not, which part do you reject?

The size of the numbers is irrelevant. I personally would consider a world in which I had no mote of dust and someone else was tortured as a world with less utility than a world in which I had a mote of dust in my eye. I expect every single one of those 3||3 individuals to feel the same way. Therefore, there is no number of individuals that I would choose not to suffer the dust mote in preference to torturing someone…

We're not talking about you accepting a dust mite in any of the scenarios. It's always other people.

So saying that you are selfish doesn't get you out of this, as long as you do prefer the world with less other people having specks.

Where is there a discontinuity when talking about other people being tortured or specked?

Re: Rationality: From AI to Zombies

#97

Post by gwern »

Kutta wrote: post #86

I'm happy to see this in the preface (where EY looks back and details what he thinks he did wrong): "Today I would write more courteously, I think. The discourtesy did serve a function, and I think there were people who were helped by reading it; but I now take more seriously the risk of building communities where the normal and expected reaction to low-status outsider views is open mockery and contempt. Despite my m…

For those interested in Scott: his LW posts can be found at http://lesswrong.com/user/Yvain/submitted/ , his old blog at http://squid314.livejournal.com/ , his new blog at http://slatestarcodex.com/ , and a tumblr for shorter things at http://slatestarscratchpad.tumblr.com/

If you're not sure where to start, a number of his writings have been received well on HN: https://hn.algolia.com/?query=slatestarcodex&sort=byPopulari...

Re: Rationality: From AI to Zombies

#98

Post by Strilanc »

zyxley wrote: post #89
Strilanc wrote: post #58

Earlier quoted context omitted.

It's disingenuous to characterize moral disagreement as ignorance. Eliezer is aware that people think the torture is worse and of the arguments they make to justify it, he simply thinks they're wrong. Why does he think they're wrong? Probably because the resulting decision theories are exploitable or inconsistent in well understood ways when operating under uncertainty. If you prefer 100000000000000000000000000000000…

> If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. I fail to see how the first part of this sentence implies the second.

That is indeed a common place where people disagree.

The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement.

Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases.

1: http://en.wikipedia.org/wiki/Allais_paradox

Re: Rationality: From AI to Zombies

#99

Post by ikeboy »

kybernetikos wrote: post #95
ikeboy wrote: post #92

Earlier quoted context omitted.

You aren't quite grokking the difference between "huge crowd" and 3||3. How do you deal with the argument from circularity? Are your preferences circular, and if not, which part do you reject?

The size of the numbers is irrelevant. I personally would consider a world in which I had no mote of dust and someone else was tortured as a world with less utility than a world in which I had a mote of dust in my eye. I expect every single one of those 3||3 individuals to feel the same way. Therefore, there is no number of individuals that I would choose not to suffer the dust mote in preference to torturing someone…

I don't see how that's relevant to answering the circularity argument. How are you avoiding the claim that your preferences are inconsistent? At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and the jump can be arbitrarily large.

(Also, if you click on the comment you can reply).

Re: Rationality: From AI to Zombies

#100

Post by kybernetikos »

ikeboy wrote: post #96
kybernetikos wrote: post #95

Earlier quoted context omitted.

The size of the numbers is irrelevant. I personally would consider a world in which I had no mote of dust and someone else was tortured as a world with less utility than a world in which I had a mote of dust in my eye. I expect every single one of those 3||3 individuals to feel the same way. Therefore, there is no number of individuals that I would choose not to suffer the dust mote in preference to torturing someone…

We're not talking about you accepting a dust mite in any of the scenarios. It's always other people. So saying that you are selfish doesn't get you out of this, as long as you do prefer the world with less other people having specks. Where is there a discontinuity when talking about other people being tortured or specked?

I know that that is the formulation of the question, But since you're asking me to make the choice, you're asking me to inflict suffering on many extra people who would not have suffered otherwise. If what is gained by their suffering is large enough compared to that suffering that I believe they all (or nearly all) would have chosen voluntarily to accept the suffering for the benefit, then I am happy with the choice, and my best model for that is what suffering I would be prepared to undergo for what benefit. I think the confusion is between evaluating the utility between two possible alternatives (where obviously the world with fewer people suffering is better) vs evaluating the utility between two possible alternatives where those were the only two alternatives. If it's just two of many alternatives, the utility placed on a meaningful sacrifice doesn't come into it, but if it's a choice, the utility placed on a meaningful sacrifice does.
Post reply on HN