I've seen friends get labeled "brogrammer" just because they wore polo shirts instead of t-shirts with gaming stuff printed on them, and because in university they belonged to a fraternity instead of the anime club. These people don't spend all day in the office doing kegstands, they work just as hard and just as effectively as anybody else, but they are derisively labeled "brogrammer" and the depth of their skill is routinely questioned.
Above a certain age, 30-35 or so I'd estimate, not being "nerd/geek/gamer" seems to lose it's stigma as "family man" becomes a more common occurrence (but of course with that comes age discrimination).
(I believe the tie-in to sexism is fairly complex. Selection for "nerd/gamer/programmer" probably causes a gender imbalance in the industry, but I think that genuine sexism and not just prejudice against lifestyles/fashions also plays an important role. Evening out the gender imbalance would likely have the effect of loosening prejudices against lifestyles/fashions, even if those prejudices were the cause of the gender imbalance in the first place. I believe the issues are related in a more complex manner than "one caused the other".)