I agree with this, fully and unequivocally. I have to wonder if "there are ladies present" might also be construed as sexist. Maybe we just shouldn't use toilet humor in situations where we're not absolutely sure everyone will find it appropriate. Warning: offensive, but to demonstrate a point... I once interviewed a guy who decided to wear a t-shirt to the interview that read "Thousands of my potential children died…
The right for our "penis jokes" also entitles us to a moral responsibility: be "a bit" sure we can make these jokes to our audience (though I admit I don't check/care for people around whose ears have long reach, but this is another debate), which is why you ought to be a little self-conscious of what you are wearing.
On the other hand, this makes me cringe a bit... I bought this hoodie that says "Cool story bro" (obvious reference to the meme), what if I stumble upon persons who will understand this as me "perpetuating the testosterone bro-culture that is driving women away from tech"?
If we start drawing the line somewhere, how do we know someone else didn't draw it earlier or later to ours?