Master-slave is not racist, is it? Nasty social-economic relationship for sure. Do modern household slave holders, did Roman or Native American slave holders use race as the basis for slavery, or simply the fact of conquest or opportunity? Only some (very significant) slave systems used race, but it's functional distinction slavers chose other groups based on, not intrinsic. That's like saying a noose is inherently racist, many more white criminals were hung from them too over time.
Why don't left-handed people complain about being sinister? Don't they want their rights?