I find this a fascinating topic to think about because I've had so much experience with it.
Sports have been a significant part of my life ever since I was in 1st grade. I played several varsity sports throughout high school and had opportunities for scholarships in two different sports. One of those sports was golf and I find that it elicits the opposite response in a way that I hate (but also understand).
When I've been at interviews for Google, Facebook, Amazon, wherever, I inevitably talk about sports with the other undergrads that are interviewing with me. In an effort to avoid the perception that I'm some sort of country club WASP, I rarely mention that I played extremely high level golf throughout high school. I fear that it won't fit into the "culture" of various places that I've applied to and likely won't mesh with those I'm talking to. So instead, I let my "hipster"-ish clothing, love of post-hardcore metal, and interest in all books not found on the NYT Bestseller list, craft whatever perception people may have of me.
But despite this, I can talk about the NBA and NFL with the best of them and I'm getting to the point where I don't care what people think because I've found it's too hard to worry about it. If I want to go throw some iron over my head 4 times a week and people consider me a meathead, so be it. But it's a whole lot easier to have them think that than to try to hide it for ridiculous reasons.