One day I had a pretty watch on, and a male coworker said "That's such a come-hither watch". He was one of the most intelligent people I'd met, and it was painful to hear that come from him. I made an exception and tried to tell him that one might wear something pretty simply because they liked wearing it but I don't believe I got my point across.
I don't think it's a reason to quit though. Everyone suffers from several disadvantages that others might not know about. For example, the guy above once complained that some american universities (like purdue university) prefer women candidates to improve their sex ratio, and that it's unfair.
I don't want to tell you to stoically put up with this problem, but that's what I try to do.