You bring up an interesting point. But even at the peak of slavery in the US, there were "white" people who believed all people are created equal and deserve equal rights, some even helped free slaves.
I think it is more of a worldview thing which is influenced by local culture. You beliefs about answers to fundamental questions like why do people exist? What are valid authority systems and structures? What is the origin of humans? And more questions like this, makes the difference.
I believe racism is wrong because I don't believe race is a thing beyond a social classification system. I believe in ethnicities and diffrences between them, but humans share a common origin. Even if some humans have less or more capability to do certain things than others, in order to be just, one must evaluate others as individuals not groups. Even if some statistic about a group can predict behavior of all but one members of a group, prejudice against that one member is still unjust and unfair, We should treat others the way we want to be treated ourselves.
But back to the topic, I think I agree with you that why we believe privacy is important is critical to make any improvements or changes. Simply put, privacy is power. When someone deprives you of privacy against your consent (where expectation of it is readonable), they are excercising power over you. They are excecising a claim that for whatever reason, information about you now belongs to them. If all or most information about you belongs to someone, you are now their subject.
It could be as simple as a lack of understanding of the implications of privacy deprivation.