Ends Don't Justify Means (Among Humans)

Although you are correct that I am assuming a third party, it is only in the role of judging the described actions as just or unjust that a third party is assumed...namely the gentle reader - a necessary assumption in any discussion of hypothetical ethical scenarios as the example of the school children illustrates.

There is a convention to these things - one doesn't assume that the dead school children were carriers of an incurable virus and that only their death prevented a deadly pandemic etc.

As I am sure you recognize, a patriotic motivation does not make an action right - even if your assumption that the rescuers and rescued share political affiliation might often lead one to assume your holding such a belief.

too much faith in 'friendly AI'. Friendly my ass! Doesn't this guy consume Science Fiction?

Seriously, I would extend his argument to assume that all 'hardware is corrupted' as he puts it, though I think it's more accurate to say 'all software is buggy'. For this amazing AI to even exist, it probably had to be pretty damn selfish, pushing out all other potential AI's to get to the top of the heap.

Just saying.

Great response.

I did not highlight the distinction that in this class of hypotheticals that the lesser harm requires action and that the greater harm requires one to simply do nothing, to stand idly by as it were - oh my god, I can here the voice of my prof in my head from days gone by as I clarify this point. Still, the causal link remains, one can either choose to act or not (or insist that you cannot even begin to play the game as was done here). But apart from that action/inaction subtlety I have to disagree with you here, it is an accurate if not entirely straight-faced summary.

Look, you might have a good working understanding of "can't occupy the epistemological state" (oh really, why? because I haven't achieved the level of perfection of my future hypothetical self) but I find it fairly meaningless. Hint: substitue epistemological with ethical or even aesthetical to see if such an assertion becomes any more meaningful. Note: I am not saying that I am positioning myself against the "you can't even begin to play (or, I'm not playing) the game" stance or some variation thereof as my response to this dilemma would be probably something along these lines given my aversion to hypothetical thought experiments such as this which I feel contribute very little to the debates in morality and ethics.

This isn't the core point of the essay. But this is not the case surely. The essay makes many points, sure, but this chain of reasoning is I believe fairly central and although it could be excised I believe that the author formulated the whole essay this way for a reason. This post-singularity being's properties are analysed in the light of a very classic problem in philosophy. If you look at the comments you will see that a poster points out that "regular" philosophers invoke mythical beings such as 'angels' or 'ideally rational agents' which are non-tech versions of what is going on here. I don't think it's a dodge, it doesn't even seem like a dodge and the author didn't even need to point this out. Where I'm coming from is that this ground has been covered and it has been covered in language that is not obfuscated. The jargon salad does nothing more than communicate "look at me, I'm so clever" which is why I claim that the author sounds smarter than he actually is.

just a way of rephrasing the idea that philosophical hypotheticals are actually useless by virtue of being impossibly overspecified which itself comes from impossible oversimplification This would be something a logical positivist would say. It's something I'm very inclined towards. I agree that hypotheticals like this generate a good amount of noise and heat but they fail to be constructive or advance our understanding of ethical questions beyond perhaps showing what ethical norms a person subscribes to, to whit: all life is sacred and one is commanded by a supreme being to do no harm, all life has intrinsic worth/value so you shall never through action do harm, you shall optimize for the greater good, and so on and so on.

Did you miss the bit where Jesus asserted power over death and came back to life after three days? The sacrifice was a blood sacrifice, not a soul sacrifice. Eternal suffering doesn't come into it.

> but I find it fairly meaningless

Why? I don't find your hint helpful; I have a pretty good idea what an epistemological state is but no idea what an "ethical state" or "aesthetical state" would be.

The meaning seems perfectly clear to me: human nature, with all its cognitive biases and imperfections of memory and perception and limited thinking speed and imagination and so forth, makes it very unlikely that you will ever really be in the situation of having to choose between definitely-for-certain killing one person and definitely-for-certain letting one die, with definitely-for-certain no other options.

Now, that isn't (as jerf already pointed out) EY's actual argument, it's a hypothetical argument he put in someone's mouth and described as a dodge. He is, I think, endorsing something along the same lines:

It is unlikely that you will ever be in such a situation and, empirically, situations at all like that are very rare. So quite likely, even if you think you are in such a situation the probability that you actually are is low. On the other hand, you're extremely likely to encounter situations where you have the opportunity to harm people while convincing yourself you're doing good overall.

Accordingly, it may very well be that net expected utility is optimized by having you follow principles like "the end justifies the means". Even when it seems to you that you're in an exceptional case where you shouldn't. In other words, consequentialists should sometimes behave like deontologists.

But some hypothetical superintelligence (EY isn't AIUI talking about a perfected future hypothetical self, by the way) might well have a much better ability to tell what situation it's in and what options it has, and much less tendency to be corrupted and self-deceiving in the same ways as we are. If so, it would not be appropriate for it to operate on the principle that the ends don't justify the means -- at least, not if the ultimate goal is to maximize net expected utility. Consequentialist AI-designers might not do best to program their AIs to act like deontologists.

I have no idea what makes you think that EY's aim is to say "look at me, I'm so clever"; for what it's worth, I find his argument clearer and less word-salad-y than yours. (Especially the weird paragraph about Jesus.)

And no, your purported summary is not accurate, for at least the following reasons: (1) EY didn't say anything about hypothetical future versions of himself, and (2) as jerf pointed out EY said in so many words that the "I refuse to answer your question because I couldn't in that epistemological state" response is "a dodge" and that one ought to have better answers to such questions. (Though he does think -- as AIUI you do too -- that this specific question may not deserve a more serious answer.)