That's how human morality works unfortunately.
capitalism: "an economic and political system in which a country's trade and industry are controlled by private owners for profit, rather than by the state."
meritocracy: "government or the holding of power by people selected on the basis of their ability."
morality: "principles concerning the distinction between right and wrong or good and bad behavior."
Here's how I think about "us looking down on them":
Assume both capitalism and meritocracy. Then the question becomes: How should superiors (in profit and ability) treat inferiors? That's the morality question. Try putting yourself in the cafeteria worker's shoes (i.e., you do not have ability and profit). How should you be treated? Surely things other than ability and profit should matter. We think "good" people deserve "good" things. What makes a person "good"? Is someone good if they were born with and then cultivated ability? Doesn't the low wage worker also work hard? Then the differences for ability are determined at birth. Does something determined at birth concern one's morality in the present? How about profit? Does someone's wealth factor into their morality?
I do not think morality is concentrated in the powerful. I think there are immoral powerful people and moral powerless people. I think it is fair to treat a moral person with respect regardless of their power position. I cannot tell whether a person is moral by their ability and profit so, a priori any other evidence, I believe in being respectful to everyone.
tldr: I judge people based on their morality and power isn't a signal. Therefore, I believe it is wrong to look down upon a person based on their power position.