That assumes that the person's work has an inherent value, and further that this inherent value is equal to the highest amount it can be paid in any market. That hardly seems reasonable.
Consider two countries, A and B. A has limited economic development, low technology, poor infrastructure, low education. B is far more advanced. A and B lie on opposite sides of the globe and have essentially never had trade or direct contact. (This is an accurate scenario before the Age of Exploration, where travel was far more limited than now, and there were already large differences in the relative level of development of various lands.)
Person Pa mops the floors in country A and gets paid $0.25/hr. Person Pb mops the floors in country B and gets paid $5.00/hr. (You may wonder how we can define both of their incomes in dollars if the two countries have no contact; you can define a virtual exchange rate based on respective purchasing power, for instance).
Your interpretation states that there is a transfer of wealth from person Pa to... person Pb? Everyone in country B? Everyone in country A? It's not clear, but any of those conclusions seems unreasonable. There is no causal connection between country A and country B, so there can be no wealth transfer between them. And how can the existence of country B, at the unknown and unreachable recesses of the earth, cause a wealth transfer within country A?
The more reasonable interpretation is the opposite: it is person Pb who is receiving a transfer of extra wealth. From whom? From the past and present inhabitants of country B, who have created the wealth, technology, infrastructure and knowledge that allows Pb to receive more for the same amount of work. This interpretation, unlike yours, is backed by clear causal connections.
I do agree, however, that people often fail to account for this transfer of wealth.