Heh, privilege, the way leftist flagellate see it is: If your life is better than someone else's, you are privileged. It is relative apparently. It is usually used to reinforce guilt and self-doubt in an attempt to shame you into caring about the plight of others. It is a devious ploy and attack on the self. You are right to question it.
Strictly speaking, by the definition of privilege, they're right in that you are in a position that not all can attain in practice. Whether it's because you weren't born a crack baby, or because you weren't implicitly discriminated against obviously for some historically known reason (sex, race, handicap).
But life isn't fair, and never will. Others will argue that they will continue to level the playing field, but there will those who are still privileged and those who aren't, for things like being born better looking, or being smarter, or being at the right place or the right time.
Ultimately people chaff against the label of 'privilege' because it has such a negative connotation. And implies that life was easy for you to begin with. You weren't privileged then, but now you are as it were.
For those that call you that, most want you to have the perspective that your life is better than others and that by noblesse oblige you have a duty to the less privileged, by some moral code be it Christian, Bushido, Chivalry what have you.
Consider it a social tax on your success.