It's not necessarily about who's turn it is, or what's "just". It's about driving rules that are crystal clear, that will end up maximizing traffic flow, clarifying r-o-w order, and minimizing time of conflict.
>Not my fault I don't want to go 20+ over.
No, but I think that people see it this way is part of the problem. It becomes a sort of moral game of chicken, with one side thinking one rule is more important, and vice versa. It shouldn't be debatable. Both sides should automatically know who has r-o-w without thinking, even if the rule is arbitrary. The entire system operates around this premise.
And because speed is always going to be a personal sliding scale (maybe 5 over is ok but 20 isn't), the simplest, most practical, consistent resolution (whether "fair" or not) is to yield the lane to the passer behind you.
Look at it this way: You wouldn't worry about momentary speed to pass someone on a two lane road, if you needed to. You'd focus on getting past and getting over before the oncoming traffic gets to you. It should be the same way on multi-lane highways, even if the traffic is overtaking instead of oncoming. That's why it's called the passing lane.