I've had many of the same thoughts. To the point that I think I've become
more racist because of them. Isn't that what racism is? Treating someone different because of their skin color?
I've instead resolved to think and worry less about it. I know I don't treat people different because of skin color, and that's good enough. If I'm rude to someone and they think it's race or gender related, that's their problem, not mine. (And I'm generally not a rude person, so it's not like happens all that often anyhow.)
What really started me down this path was a friend of mine (she's a "person of color") who honestly believes that "all white people are racist", "it's impossible to be racist against white people", and "all cops are corrupt", among other things.
After a while, I started to worry that all non-white people thought everyone was being racist all the time. Her poison infected me to the point that every time I passed a co-worker, I'd wonder if thought I was doing something racist at that moment.
I wasn't. I knew I wasn't. But I worried that they thought so.
But unless they're really good at hiding those feelings, they aren't thinking that. I know some of them well enough to know they'd show some outward sign of discomfort if that were the case.
In the end, I decided my friend was toxic to her own cause. Her constant (racist) posts about racism actually pushed me in the wrong direction, and I've had to correct for it.
Instead, I recognize that it's a kind of mental illness to expect everyone to be racist to you. Not just to guard against it, but to actually expect it. It's also a mental illness to think that nobody else could experience what you do, or understand it without actually experiencing it.
I can't fix those people. I can't only walk away and let them think whatever they want. I'll keep being a good person, and they'll have to deal with any bad feelings they're creating out of nothing.