If you are a woman you can easily counter this behavior by labelling it and saying that you don't have a porcelain skin. Bonuspoints if you laugh about a guy giving you super bad feedback and how this did not bother you. Putting people at ease around you (especially customers) is a critical entrepreneurial skill. You can't blame people for being cautious when a lot of people are buying into victim-narratives and conv…
And even if one were to believe that that is unlikely too, nothing guarantees that someone else won't think it sexist. For example, I remember some panel with four scientists (three men and one woman) that was discussed in HN a while ago; at some point, someone in the audience (I think she was a journalist) yelled at the moderator to "let her speak"... Even though the scientist herself didn't think the moderator was doing anything wrong.