While this is a simplistic view that might help you feel good about taking advantage of someone, it is ridiculously easily abused. And, quite frankly, scares me if I think about it for too long.
I don't want a society where there are people left behind simply because they did not make a job out of making more money. Nor do I want a society where services performed by folks leads to a massive wealth imbalance in workforces that don't have people specifically tasked with keeping this from happening. Especially with how anti union/similar setup current cultures are.
And to be clear, I realize that that ship has somewhat sailed. Nobody was specifically tasked with making sure women/minorities were paid fairly. As a result, we have an obvious problem. One where the incentives have aligned such that many companies do not have an incentive to fix it. (There has to be a term for the occurrence where virtue signaling is high, but actual virtue is punished. Anyone know it?)
That is, you are absolutely right in many senses. But I worry that as long as not all companies are paying more fair wages, than there is little incentive to be the one company paying fair wages. (There are incentives to pay above normal pay, but that can still be below fair. See again current women's pay gap.)
So, can you reduce it down to folks are ultimately paid what they are worth. And if they simply shopped themselves around they could get more, but they didn't so they have accepted their low worth, so they are, ipso facto, worth less. Great. I just think we have plenty of evidence that shows that doesn't actually work.