I came here to say something similar: that the same action or speech can/will be interpreted very differently based on a whole slew of factors, including gender, ethnicity, or just plainly being different people to begin with. What appears "okay" or "benign" or "in jest" can certainly not be the case for a colleague, whichever gender they may be [1].
In fact, I'd argue that it's hard to perceive the inappropriateness of our own actions in the moment. It's much easier to notice the possible auxiliary effects of our past actions when reflecting on them at a later time. Frankly, I can think of any number of things that I've said or done in the recent past that may have made my female friends and colleagues twitch an eyebrow, but didn't elicit an explicit response because they've learned to just deal with it and let it go, having been in a boys club for so long (one started her career in investment banking). And I'd consider myself to be fairly self-aware of these issues, and yet I've admittedly failed many times over.
edit: and frankly, how many of us voluntarily and regularly look back at our past week/month to think about the ways in which we've failed ourselves and our own principles in this area? We may review the ways in which we failed in our job capacity and reflect on how we can improve, but I think the frequency with which we do the same for our social interactions is much lower.
[1] In fact, I think "I was just kidding" is one of the lamest excuses for actions that at times clearly are not thus.