Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rationality: From AI to Zombies

intelligence.org

Post reply on HN
Print view View original

101–110 of 142 posts

Re: Rationality: From AI to Zombies

#101

Post by kybernetikos »

ikeboy wrote: post #99
kybernetikos wrote: post #95

Earlier quoted context omitted.

The size of the numbers is irrelevant. I personally would consider a world in which I had no mote of dust and someone else was tortured as a world with less utility than a world in which I had a mote of dust in my eye. I expect every single one of those 3||3 individuals to feel the same way. Therefore, there is no number of individuals that I would choose not to suffer the dust mote in preference to torturing someone…

I don't see how that's relevant to answering the circularity argument. How are you avoiding the claim that your preferences are inconsistent? At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and the jump can be arbitrarily large. (Also, if you click on the comment…

There's a difference between the utility of a world where one person has a mote of dust in their eye and the utility of a world where one person has a mote of dust in their eye because it saves someone else.

>At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and the jump can be arbitrarily large.

Well, the boundary is fuzzy, so I believe that different people will draw the line in different places, but yes, that is correct. There is a point at which I accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, and that point is the point at which I determine that the hurt falls under the threshold I would expect every person to be prepared to sacrifice for any other person.

I think the debatable range is actually quite large, so the tiny decrease part might not be fair (there are a lot of degrees of pain I would not demand someone to suffer to save others), but at that point of expected sacrifice, the number of people jumps to infinite.

Re: Rationality: From AI to Zombies

#102

Post by zyxley »

kybernetikos wrote: post #101
ikeboy wrote: post #99

Earlier quoted context omitted.

I don't see how that's relevant to answering the circularity argument. How are you avoiding the claim that your preferences are inconsistent? At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and the jump can be arbitrarily large. (Also, if you click on the comment…

There's a difference between the utility of a world where one person has a mote of dust in their eye and the utility of a world where one person has a mote of dust in their eye because it saves someone else . >At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and t…

> and that point is the point at which I determine that the hurt falls under the threshold I would expect every person to be prepared to sacrifice for any other person.

That seems like a good way to put it.

The number of people doesn't really matter, when it the amount of hurt per person is so small that you can say with all confidence something like "I believe that literally any sane person would agree to get one speck in their eye to save a stranger from fifty years of torture".

Re: Rationality: From AI to Zombies

#103

Post by zyxley »

Strilanc wrote: post #98
zyxley wrote: post #89

Earlier quoted context omitted.

> If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. I fail to see how the first part of this sentence implies the second.

That is indeed a common place where people disagree. The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things th…

> The basic reason it should follow is that not doing so makes you inconsistent and exploitable

I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?

Re: Rationality: From AI to Zombies

#104

Post by Neslit »

unbudgingprawn wrote: post #93
kenbellows wrote: post #71

Earlier quoted context omitted.

In attempting to debunk this sort of statement, he has literally written the following: "In the hands of RationalWiki generally, and RationalWiki leader David Gerard particularly ..., [the fictional version of the Roko's Basilisk event] somehow metamorphosed into a Singularity cult that tried to get people to believe a Pascal's Wager argument to donate to their AI god on pain of torture. This cult that has literally…

"This sentence is a lie." He doesn't seem to actually debunk the idea, just yells about David Gerard for a bit. Can you explain how it's different from Pascal's Wager?

At least the part where it's implied that "Someone from organization X used the basilisk to convince innocent bystanders to donate to X" is false, for any value of X that I know of. Pascal's Wager is/was (I imagine?) used to get people to subscribe to various churches and give them money.

Re: Rationality: From AI to Zombies

#105

Post by zyxley »

convexfunction wrote: post #43
Retric wrote: post #31

Earlier quoted context omitted.

People tend to ignore that for a logical trail to work every stated and unstated assumption needs to be 100% true or it's pointless. Roko's Basilisk is great example where even the slightest weakness in any part of the argument makes the whole idea pointless. (And there are a lot of unstated assumptions.) EX: What's the maximum computational power of the universe?

Eliezer agrees with you! Roko's Basilisk in particular is wrong. As a matter of policy though, you do not just publicly publish information hazards that might be real ; that was why it was a big deal.

> you do not just publicly publish information hazards that might be real

But... for "might be real" to be valid at all, you have to buy into the Roko's basilisk concept in the first place.

If you don't, there's nothing to get upset about.

Re: Rationality: From AI to Zombies

#106

Post by zyxley »

ikeboy wrote: post #94
pyrois wrote: post #73

Earlier quoted context omitted.

He's assuming linearity, which at the very least needs justification. He's assuming that the function that maps from the pair (number of people, type of torture) to suffering is linear in the number of people, and also linear in the type of torture. I don't believe either of those things are true. To put it more clearly, he says: > So we can keep doing this, gradually - very gradually - diminishing the degree of disc…

It doesn't need to be linear, just not slowing super-exponentially. At almost any slowdown rate, specks will still end up worse than torture.

But that's assuming that unpleasantness per individual is a continuous function always distinguishable from normal human experience, rather than something like a line eventually rising from the discontinuous murky soup that contains all the constant minor annoyances of being human (itchy nose, wedgies, the feeling of the back of your tongue against the roof of your mouth, one shoe being a little tighter than the other, etc) that the brain is well-optimized to tune out and rapidly forget about after the fact.

Re: Rationality: From AI to Zombies

#107

Post by Strilanc »

zyxley wrote: post #103
Strilanc wrote: post #98

Earlier quoted context omitted.

That is indeed a common place where people disagree. The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things th…

> The basic reason it should follow is that not doing so makes you inconsistent and exploitable I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?

I'm not trying to put forth a framework, I'm trying to explain the side you're not on without straw-manning them.

- If you prefer 100000000000000000000000000000000000000 specks to 1 torture

- and you follow the axiom that A > B implies x% A > x% B, and value people equally

- then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture

- and you will want to throw a speck in someone's eyes whenever it decreases the odds of torture by 0.000000000000000000000000000000000000001%

- any given person has a more than 0.000000000000000000000000000000000001% of going nuts and trying to torture someone in the next day

- and you can decrease those odds non-negligibly by continuously throwing specks into their eyes... so do that

I will grant you that no person would take that last step. It's too stupid, too not-common-sense. But code... code does whatever you tell it to. And if you tell it to prefer 100000000000000000000000000000000000000 specks to 1 torture, and to follow decision theory axioms, don't be surprised when the solutions it spits out uses specks of dust to prevent people from seeing for absurdly small chances of preventing torture.

Re: Rationality: From AI to Zombies

#108

Post by ikeboy »

kybernetikos wrote: post #101
ikeboy wrote: post #99

Earlier quoted context omitted.

I don't see how that's relevant to answering the circularity argument. How are you avoiding the claim that your preferences are inconsistent? At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and the jump can be arbitrarily large. (Also, if you click on the comment…

There's a difference between the utility of a world where one person has a mote of dust in their eye and the utility of a world where one person has a mote of dust in their eye because it saves someone else . >At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and t…

So your intuition says that there's some point where everyone is suffering pain X, where X is very large, and they would each agree to cause the X-(1 mote speck) to a trillion times as many people, in return for reducing their own suffering by 1 mote speck? That strikes me as beyond regular selfishness, and non intuitive.

Re: Rationality: From AI to Zombies

#109

Post by ikeboy »

zyxley wrote: post #102
kybernetikos wrote: post #101

Earlier quoted context omitted.

There's a difference between the utility of a world where one person has a mote of dust in their eye and the utility of a world where one person has a mote of dust in their eye because it saves someone else . >At some point, you need to accept a huge jump in the number of people getting hurt in return for a tiny decrease in the amount of hurt for each one, where the decrease can be pretty much arbitrarily small and t…

> and that point is the point at which I determine that the hurt falls under the threshold I would expect every person to be prepared to sacrifice for any other person. That seems like a good way to put it. The number of people doesn't really matter, when it the amount of hurt per person is so small that you can say with all confidence something like "I believe that literally any sane person would agree to get one sp…

So first of all, your preferences are still circular, unless you bite another bullet somewhere. But even your argument isn't quite accurate. It's not this one person who needs to get a speck, it's a literally unimaginable amount of people.

As it happens to be, a number of people in our tiny world have said they would choose torture, so your argument fails just considering them.

Re: Rationality: From AI to Zombies

#110

Post by ikeboy »

zyxley wrote: post #106
ikeboy wrote: post #94

Earlier quoted context omitted.

It doesn't need to be linear, just not slowing super-exponentially. At almost any slowdown rate, specks will still end up worse than torture.

But that's assuming that unpleasantness per individual is a continuous function always distinguishable from normal human experience, rather than something like a line eventually rising from the discontinuous murky soup that contains all the constant minor annoyances of being human (itchy nose, wedgies, the feeling of the back of your tongue against the roof of your mouth, one shoe being a little tighter than the othe…

Even if it's discontinuous, it still needs to grow incredibly slowly. The ratio of pain neurons that fire for a speck versus for torture over 50 years is nowhere near 3||3. You need a function that grows so slow that outweighs 3||3, which is damn near impossible for any plausible function.
Post reply on HN