This is a good point, but a lot of ressources have a fixed or limited supply (arguably all of them); if wealth inequality increases, the poor fraction of the population will have a harder time competing for those.
Consider urban housing as an example (specifically price development in terms of median income, and how the supply side reacts to wealth distribution by "overdelivering" luxury appartments from the average citizens point of view).
Increasing inequality is also problematic because it fosters rent-seeking behavior which is self-reinforcing (because this siphons income from the poor side of your distribution to the wealthy one).
It might well be better to be less wealthy in a society with lower spread.
You could also argue that most wealth right now is accumulated/grown by "extracting" a bit of the value from the work of others. Consider Valve (the game distribution platform) for a very obvious example: They make something around $50M per employee in revenue. Are their employees working ten times harder than average game developers (by literally any reasonable metric)? I'd argue that their company became very good at extracting value from the whole market, instead. Absurd wealth does not come from doing lots of work yourself, it comes from taking a little bit from lots of people.