Because privilege is based on looking at only a subset of the group. A subset that changes to push a narrative. For example, white privilege often involves some concept related to being given leniency in legal matters. Less likely to be stopped, less likely to be arrested if stopped, less likely to be charged if arrested, and less likely to be convicted if charged. And statistically, that is all good and fair. But then male privilege discussions almost never include that the reverse happens when you look at gender (and to a much greater degree than race).
Also, while perhaps not originally meant to be applied to an individual, that is the common experience with it. It is further compounded by picking only certain areas for privilege to matter and being very stereotyped. For example, one white privilege is going to a school where the majority of the student body was the same race. Except that isn't always true.
While the wording behind privilege is extremely toxic and needs to be done away with, much of the ideas behind it are what directly led to that toxicity to begin with.
>THEY ARE NOT REFERRING TO YOU AS AN INDIVIDUAL FOR THE LOVE OF GOD
Yeah, if you ignore all the other times.