Now imagine that instead of a mugger you have an AI researcher who both believes that AGI will destroy the world and is intent on being in the room when it is created. The self-mugging of an AI safety-ist.
Bentham's Mugging (2022)
81–90 of 103 posts
Re: Bentham's Mugging (2022)
#82The problem here isn't with the main character's moral philosophy, but with his decision theory. He'd be dealing with exactly the same predicament if the mugger were threatening to harm him . The solution is indeed "don't give in to muggers", but it's possible to define this in a workable way. Suppose the mugger can choose between A (don't try to mug Bentham) or forcing Bentham to choose between B (give in) or C (don…
The mugger in the story is essentially contriving a situation that turns him into a utility monster. He is arranging that he will derive more benefit from the money than any other plausible application -- by imposing a massive harm on himself if he doesn't get the money. It's relatively straightforward to vary the threat to adjust incentives as necessary -- e.g. the binding deal with the thug later in the story.
One related way to think about this is that, if (non-utilitarian) people are ever able to change their own utility functions, other people's utilitarianism gives them an incentive to do so, because it can make them more likely to get what they want.
For example, if you can learn to feel sadder about something, other people who want to minimize your sadness will acquire an incentive to help you avoid that thing, even at some cost to themselves.
In many moral intuitions, you find the world as it is and then act on it in some way, without other people strategizing about, or being incentivized by, your moral reasoning. But when other people can do those things, you can get very weird outcomes.
Re: Bentham's Mugging (2022)
#83The underlying assumption is that Bentham is a true act utilitarian yet simultaneously has 10 pounds in his pocket that he can stand to lose without much harm. If he truly were an act utilitarian, the utility of the 10 pounds remaining in Bentham's possession must be so high that it outweighs the mugger losing their finger, otherwise Bentham would have already spent it on something similarly utility maximizing. Clear…
It's for this reason that I find all forms of Utilitarian less edifying than I would hope. They can put intuitively moral choices on a clear, explicit footing, but they produce counterintuitive results in less clear cases. Those less clear cases are precisely what you'd want a Utilitarian system to solve, since you don't really need to make much effort to justify clear cases. They tend to look best on the moral equiv…
> They tend to look best on the moral equivalent of PowerPoint slides.
This is an interesting formulation, when you put that way I imagine most readers have seen some 'moral powerpoint slides' that fall apart on a closer examination.
Re: Bentham's Mugging (2022)
#84Earlier quoted context omitted.
I'm not sure that SBF being a crook shows that effective altruism failed.
One main idea of EA is that you should make a lot of money in order to give it away. The obvious problem is that this can serve as a convenient moral justification for greed. SBF explicitly endorsed EA, Will MacAskill vouched for him, and I understand he was widely admired in EA circles. And he turned out to be the perfect incarnation of this problem, admitting himself he just used EA as a thin veil. What would you c…
If effective altruism was not much good for the average adherent I'd say that was a fail. SBF was an outlier.
Re: Bentham's Mugging (2022)
#85The underlying assumption is that Bentham is a true act utilitarian yet simultaneously has 10 pounds in his pocket that he can stand to lose without much harm. If he truly were an act utilitarian, the utility of the 10 pounds remaining in Bentham's possession must be so high that it outweighs the mugger losing their finger, otherwise Bentham would have already spent it on something similarly utility maximizing. Clear…
I get what you mean. Whenever I read one of these scenarios that tries to show some absurd conclusion about a moral theory like utilitarianism, it makes me wonder if there is something about the outlandishness of the situation that makes it hard for our mental heuristics to work properly. In this case it's the idea that Bentham is a perfect act utilitarian, that the mugger will 100% follow through on his promise, that it won't affect anything else in the future besides the immediate suffering etc.
That said, I am doing my best to come up with an example that avoids the problem you mention. If we can imagine a utopia society where everyone is a perfect act utilitarian except the mugger, and all resources are distributed in a totally fair way such that any $10 will buy much less utility than saving a finger, the mugger's tactic would be harder to avoid.
I think the problem I still have with this is that it's basically saying that it's possible for a jerk to take advantage of a bunch of nice people, which isn't that interesting of a conclusion.
Re: Bentham's Mugging (2022)
#86Earlier quoted context omitted.
My wording is wrong, because it sounds like I'm saying that Bentham is adopting the policy ad hoc. A better way to state this is that Bentham starts out as an agent that does not give into brinksmanship type games, because a world where brinksmanship type games exist is a substantially worse world than ones where they don't (because net-negative situations will end up happening, it takes effort to set up brinksmanshi…
> It's different because by adopting C, Bentham prevents the mugger from mugging, which is a better world than one where the mugger goes on mugging. This assumption is wrong. You are assuming that the mugger is also a utilitiarian, so will do cost-benefit analysis, and thus decide not to mug. But that is not necessarily true. If the mugger mugs anyway, despite mugging being "suboptimal," Bentham ends up in a situatio…
Except that eventually the mugger will run out of fingers (and/or reattaching them will eventually start not working out), so the mugger will be forced to stop mugging. Well, ok, they could start cutting off toes, or threaten some other form of bodily mutilation.
But regardless, giving in to the mugger enables the mugger to continue mugging indefinitely. Not giving in -- assuming the mugger goes through with whatever self-mutilation they've threatened -- will eventually cause the mugger to stop mugging. This would be a net positive, better than allowing the mugger to mug indefinitely.
On top of that, I was disappointed that, in the story, Bentham does actually bring up the idea that capitulating could encourage copycats to run similar schemes, but this rationale for not cooperating is hand-waved away. This is pretty standard "don't negotiate with terrorists" stuff. Giving in just tells the mugger -- and other potential muggers -- that this strategy works. Surely it's more utilitarian to stamp this out at the source, even if it costs the original mugger some fingers.
(But I guess this is in part the point of Bentham being an act utilitarian in the first encounter, as he wouldn't consider the larger implications of his actions, just on the specific, immediate result of the action in front of him.)
Re: Bentham's Mugging (2022)
#87I feel mugged having attempted to read that nonsense ;-)
Re: Bentham's Mugging (2022)
#88Earlier quoted context omitted.
> It requires a total ordering and an aggregation function Yes. And note that this is true even for just a single person's utilities, i.e., without even getting into the issues of interpersonal comparison. For example, a single person, just to compute their own overall utility (never mind taking into account other people's), has to be able to aggregate their utilities for different things. > if both exist, then there…
Can you explain the "for any reasonable treatment" part here? How is the least upper bound property (or something equivalent to it) getting used in quantitative ethical theories?
Because even if we assume that we can assign numbers to a single person's utility for many different things, we cannot assume that the ratios between these utilities will be integers or rational numbers. We have to consider the possibility of utility ratios that are irrational numbers. In treatments like that of Von Neumann, where numerical utilities are assigned meaning by asking hypothetical agents to accept or decline a potentially infinite series of bets, the possibility of utility ratios that are irrational numbers arises as the least upper bound property requirement--because the final utility ratio from the infinite series of bets will be the least upper bound of the series of ratio estimates that arise from the bets.
Re: Bentham's Mugging (2022)
#89Earlier quoted context omitted.
Can you explain the "for any reasonable treatment" part here? How is the least upper bound property (or something equivalent to it) getting used in quantitative ethical theories?
Because even if we assume that we can assign numbers to a single person's utility for many different things, we cannot assume that the ratios between these utilities will be integers or rational numbers. We have to consider the possibility of utility ratios that are irrational numbers. In treatments like that of Von Neumann, where numerical utilities are assigned meaning by asking hypothetical agents to accept or dec…
Thanks!
That also raises a different question about people's introspection and ability to accurately answer questions about their utility and preferences, but that's a bit far afield from the mathematical question.
Re: Bentham's Mugging (2022)
#90The underlying assumption is that Bentham is a true act utilitarian yet simultaneously has 10 pounds in his pocket that he can stand to lose without much harm. If he truly were an act utilitarian, the utility of the 10 pounds remaining in Bentham's possession must be so high that it outweighs the mugger losing their finger, otherwise Bentham would have already spent it on something similarly utility maximizing. Clear…
I get what you mean. Whenever I read one of these scenarios that tries to show some absurd conclusion about a moral theory like utilitarianism, it makes me wonder if there is something about the outlandishness of the situation that makes it hard for our mental heuristics to work properly. In this case it's the idea that Bentham is a perfect act utilitarian, that the mugger will 100% follow through on his promise, tha…
> scenario that tries to show some absurd conclusion about a moral theory
So, the trolley problem? "Philosophy is bunk".