I am in no way worried about internet points. My concern is trying to parse social signalling. Downvotes can express rebuke of undesirable behavior here. Ignoring that fact is not a good thing.
I am very leery of concluding that downvotes can be wholesale ignored simply because I am a demographic outlier and many people will knee jerk disagree with me. That fundamentally breaks an important social feedback loop.
I didn't make it to the leaderboard by assuming I was getting pushback merely because I am female and men here are clearly sexist pigs and I can ignore their rebuke. I put quite a lot of time and effort into sorting out what I could ignore and what I needed to take seriously as an indicator that I needed to do something differently.
Granted, what I did different wasn't necessarily what other people thought I should. But that social feedback loop matters. It never works well to suggest that minorities can and should simply wholesale ignore such feedback. That goes bad places.
It has been extremely helpful to me that really ugly comments aimed at me are often flagged to death. Unfortunately, this sometimes only happens if I point it out. Then my comment pointing it out is often also flagged and/or downvoted.
It has also been extremely helpful that some people give corrective upvotes. I, also, do that to comments that I don't think really deserve to be in the negatives. Getting back to even after being downvoted feels much, much more to me like "Okay, others don't see it that way and your viewpoint is an outlier for this forum" and much less like "Shut up, bitch. No one wants you here anyway."