Rationality: From AI to Zombies

I have edited that post to recommend the book over my ePub. You can always read my compilation after finishing the book, if you want to read the articles the book leaves out.

If I learned that a co-worker believed that nuts were invariably fatal to me and gone on to hide some in my food, the nuts would do me no harm but I would still take the whole incident seriously.

He went back and hid the more sensitive LJ entries, IIRC, so I don't think it should be a problem.

This is not really a good format for continuing this discussion, because lack of notifications and such. Would you consider opening an account on lesswrong and posting in open thread? Or you could PM my account there at http://lesswrong.com/user/ike/overview/. That said, here's my reply.

>Can you explain more clearly why what I said implies that, because I don't think I mean to say that.

It's basically a reformulation of the circularity argument.

I assume there's some level of pain that you would prefer to the specks; say a single second of torture, equivalent to a smack or such. (If you think we should prefer 3|||3 specks to one smack, I could go further, so let me know.)

So counting up from that one second at a time (i.e. 2 seconds of pain, 3 second, etc), eventually we reach a point where you no longer think it's better than the specks. Call this X.

So X and X-epsilon are qualitatively different; the lower amount is not bad enough to outweigh specks, but the higher amount is. You need to prefer giving X-epsilon to a large number of people rather than X to a single one, if the qualitative difference is to be upheld.

(I may not be phrasing this so well. Maybe try working through the circularity argument above, or the other phrasings I used in this thread.)

Now to respond to your line of reasoning: this proves too much.

Imagine instead of dust specks, we want everyone to donate a dollar to save the person from torture. Are you really going to say that we should be spending unbounded amounts of money (3|||3) to save anyone from torture? Have you donated all the money you could get to prevent torture? (and yes, I'm sure there are charities that are at least partially effective.)

Why doesn't your argument work for the case I just outlined as well?

>Let's say you tell those 3|||3 who wanted to save someone from torture by accepting a speck in their eye that they cannot, and someone must be tortured instead. You've massively increased the unhappiness in the world - not only is an individual getting tortured, but 3|||3 have ended up with a situation that's worse than they wanted. Are you going to claim that it's still got a higher utility? Now that you notice that you're making those 3|||3 unhappy by the choice, you can see that the disutility scales with the number of people - that's why the number of people becomes irrelevant

If the people are told of the choice, that's a whole new problem, but that's kind of avoiding the point of the original question. To use a hacking analogy, you're using a side-channel to cheat.

Nobody is told about any of this. If they were, that would itself need to be factored in, and quite possibly lead me to prefer specks.

>If it's a negligible fraction that scales, and I could plausibly think that any sane person should donate that fraction of their money to save a person from torture then yes.

Each person isn't donating to save someone from torture, they're donating to save 1/3|||3 of torture.

Let's rephrase the original question to zero in on that last point. There are 3|||3 people. You choose the number of people who donate one dollar, which can be any number between 0 and 3|||3. After you make a decision, one of those people is chosen at random, and they are tortured iff they did not donate.

If you think about it, this results in the exact same outcomes in either choice, except you have more options than all or nothing. You basically choose the probability of torture.

To be consistent with your previous view, you'd need to not pick zero donaters. So for at least some people, it should be worth it for them to pay 1 dollar to avoid a 1/3|||3 probability of torture.

This dissolves, as you may have noticed, into Pascal's Wager. (Or Pascal's Mugging, more precisely, which was coined by Yudkowsky.) What if I tell you that unless you give me a dollar, I'm going to torture you for 50 years? The probability of that being true is more than 1/3|||3 (and if you disagree, then you are way too overconfident for life. 3|||3 is a huge number, and there's no conclusions I can think of in which I'd place that much confidence. We don't even have anywhere near the kind of raw data to draw any conclusion that confident.) So are you willing to give up a dollar to avoid the >1/3|||3 chance that I'm telling the truth?

If not, and all or most of those people in the problem would answer the same, then your previous rationale falls apart.

(Oh, and this is not the real Pascal's Mugging; that's much harder to deal with. But let's stick with the easy stuff for now, shall we?)

Also, what about the argument in http://lesswrong.com/lw/kn/torture_vs_dust_specks/ueo

(And if we're going to continue this, could you please move it over to another forum? It's getting harder to keep track of comments here.)