Just because some of the people you disagree with are muddleheaded doesn't mean that all of them are. And just because you see an imprecise description of a behavior doesn't mean that behavior does not exist.
"Racism" is a catch-all for the typical set of human behavior directed towards out-groups. This includes distrust, aggression, and in-group signaling. It doesn't matter what defines out-group: dress, gender, skin color, etc, but is generally something tied to identity, something easy to recognize and hard to change. Negative feelings get amplified and applied to all members of the group while positive feelings are suppressed.
In its most extreme form, this permission-to-otherness motivates genocide. Slightly farther down, slavery. Waaaay at the other end it manifests as nervousness around Koreans or something. This is just human nature. There are quite a few things in our nature we no longer casually tolerate in society and work to eliminate. Like ignorance or incest.
But on top of that, context matters. In a situation with heavy power asymmetry, both protagonists may harbor "racist" feelings, but the only attitude that matters to outcome is the attitude of the one with the power. If you feel you're being held to a higher standard, its probably because you are.
"Racism" as used, say, in the 50s, was shorthand for "institutional racism". The game had been rigged against minorities, most especially blacks, to the point that the law told them where to live and eat, who they could and could not marry, etc. This was enforced day to day by the actions of regular people employing regular, everyday racism directed downwards in a situation of power asymmetry.
Since then the institutional stuff has largely been dismantled or covered with tarps, and institutional "fixes" applied, often unevenly and stupidly, but that's also human nature.