AI recognition of patient race in medical imaging: a modelling study

Frankly, even a freshly arrived alien from Mars or Titan could easily tell Icelanders, Mongols and Xhosa apart, without knowing anything about our culture. The fact that there has been a lot of interbreeding/admixture since the Age of Sail began, does not mean that there aren't meaningful biological differences between the original groups, which still obviously exist.

An analogy: much like the existence of twilight does not render the concept of night and day a 'social construct' either. We attach certain social meanings to those natural phenomena, and a 'working day' can easily stretch into 'astronomical night' (all too often!), but that does not mean that 'night' and 'day' do not exist outside of our cultural reference framework.

There is a social concept of 'race' which corresponds to the 'working day' concept in this analogy, e.g. 'BIPOC', claiming Asians as 'white adjacent' or classifying North Africans or Jews as 'white', even though they may not necessarily look white. But this is almost certainly not what the AI identified. This social concept of race would confuse a Martian alien unless he started to study the social and racial history of the U.S., and possibly even afterwards. It definitely confuses me, a random observer from Central Europe.

It doesn't seem like noise in the images is a factor

>Race prediction performance was also robust across models trained on single equipment and single hospital location on the chest x-ray and mammogram datasets

Sure, it’s possible that bias due to the radiographer is the culprit, but this seems unlikely.

>"We found that deep learning models effectively predicted patient race even when the bone density information was removed for both MXR (AUC value for Black patients: 0·960 [CI 0·958–0·963]) and CXP (AUC value for Black patients: 0·945 [CI 0·94–0·949]) datasets. The average pixel thresholds for different tissues did not produce any usable signal to detect race (AUC 0·5). These findings suggest that race information was not localised within the brightest pixels within the image (eg, in the bone)."