The problem isn't that "privilege" has become too narrow, but that it's become too broad.
The word implies an actor. Privilege is, by definition, caused. A benefit of privilege by definition originates in a system of structural inequality, one more or less deliberately designed to benefit some, but not others, based on some criterion such as sex, skin color, sexual preference, or the like - and that's not a possibly tendentious definition of my own invention, but rather the one applied by the proponents of the concept, all the way back to McIntosh's formalization in the late 80s that began the modern popularization of "privilege" in the first place.
A further implication, also original but more clearly elucidated by Maltz Bovy, is that to do other than abandon or at least forswear the benefit of privilege, and work to dismantle the structural inequality that furnishes it, is iniquitous. To say that someone is or has been privileged therefore cannot but at minimum verge upon an accusation that that person lacks virtue. That, I think, is at base what provokes anger in response - and not unreasonably so, because it does constitute an attack on the quality of one's character. Even when leveled at someone who is progressive and does accept the theory of privilege without question, such an attack tends to elicit a defensive response. How much more so, then, from someone who is not, and does not?
And "privilege" remains so freighted, regardless of intent. It's especially pernicious, therefore, to conflate privilege with simple good fortune, which is fundamentally distinct in that good fortune isn't caused. It's just a thing that happens, and happens to work out well for someone. That's what we're talking about here. No system of structural inequality put those trees next to that hovel. You cannot reasonably posit a deliberately unequal allocation of benefit to have brought about that result. The very idea is risible at best, outright nonsense at worst. And, in stating nonetheless that that must be the case, you level the sort of opprobrious allegation I describe above.
It is not reasonable, nor is it acceptable in any conversational context where civility and charity are meant to hold sway, to offer one's interlocutor insult, then become incensed when said interlocutor responds as though insulted. Cet animal est très méchant: Quand on l'attaque, il se défend. Voltaire's sarcasm remains a sarcasm, whether you mean it so or otherwise.