> They're white men with rich parents and Ivy League degrees and so on.
I think that's a very good point.
However, that doesn't mean that racism doesn't also exist and is not very prevalent. Endless research supports that it exists in many areas of society, it seems very unlikely that after centuries of overt, legalized racism it would disappear so quickly, and IMO it's unlikely to be coincidence that the excluded groups today are exactly those that that suffered overt, legalized discrimination until the civil rights era.
Sadly, my anecdotal experience is far too many white people expressing racist views in private, often grouping all blacks together and saying 'they' do X or Y. One, who holds a PhD, told me that Kenyan runners were faster because the harsh life of slaves caused natural selection to favor strength!
I would be interested in research on the relative social mobility of various excluded groupings (people who are black, Latino, Muslim, or women) and also of poor white people. Lots of people try to claim there is no difference, as a sort of political blow against people seeking to address race issues, but I've never seen data supporting it.