Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI recognition of patient race in medical imaging: a modelling study

thelancet.com

Post reply on HN
Print view View original

91–100 of 180 posts

Re: AI recognition of patient race in medical imaging: a modelling study

#91

Post by towaway15463 »

_8j50 wrote: post #32
mensetmanusman wrote: post #4

What does this mean in terms of race being a social construct/concept?

Still is? The AI is correlating biological features with self reported race. There are biological differences between people who have different ancestors. Finns are different from brits. The spanish are different from russians. Nigerians look different than somalians. The Japanese look differnet than filipinos. Race picks specific and arbitrary differences , for example hispanic is a different race in US society but…

So isn’t the “evil social construct” part actually the invalid extension of the theory that biological or phenotypic differences mean that someone is more or less human? You can remove that part and still acknowledge that there are biological differences between people based on their genetic lineage without invalidating their basic humanity.

Re: AI recognition of patient race in medical imaging: a modelling study

#92

Post by SpicyLemonZest »

civilized wrote: post #86
SpicyLemonZest wrote: post #34

Earlier quoted context omitted.

It seems like the reason the researchers in this paper are concerned is precisely that they tried and failed to understand how the ML algorithms are doing this. If they’d discovered that white people have a subtly distinctive vertebra shape the model was detecting, it would have been much more of “oh, we discovered a neat fact”.

I don't think they tried very hard at all. I see no meaningful use of modern explanation tools. There are lots of known ways in which people of different races are different physiologically. Probably even more unknown ways. There could also be differences in imaging technology used in different communities, as others have suggested. I'd be a bit surprised if something like that could create such a strong signal but i…

For those of us less familiar with this space, what are these modern explanation tools? (I certainly agree that it's plausible the model is seeing a physiological difference, and the researchers seem to have considered a few concrete hypotheses on that dimension.)

Re: AI recognition of patient race in medical imaging: a modelling study

#93

Post by matthewdgreen »

ibejoeb wrote: post #75
matthewdgreen wrote: post #42

Earlier quoted context omitted.

"This issue creates an enormous risk for all model deployments in medical imaging: if an AI model relies on its ability to detect racial identity to make medical decisions, but in doing so produced race-specific errors, clinical radiologists (who do not typically have access to racial demographic information) would not be able to tell, thereby possibly leading to errors in health-care decision processes."

Typically? It's coded in the standard. There's a DICOM tag for it. https://dicom.innolitics.com/ciods/procedure-log/patient/001...

Unlike the authors of this research paper I am not a trained clinician, so I can't tell you. However I would note that the first exemplary value in the link you gave me is "REMOVED".

Re: AI recognition of patient race in medical imaging: a modelling study

#94

Post by civilized »

SpicyLemonZest wrote: post #92
civilized wrote: post #86

Earlier quoted context omitted.

I don't think they tried very hard at all. I see no meaningful use of modern explanation tools. There are lots of known ways in which people of different races are different physiologically. Probably even more unknown ways. There could also be differences in imaging technology used in different communities, as others have suggested. I'd be a bit surprised if something like that could create such a strong signal but i…

For those of us less familiar with this space, what are these modern explanation tools? (I certainly agree that it's plausible the model is seeing a physiological difference, and the researchers seem to have considered a few concrete hypotheses on that dimension.)

Here's an introduction to one technique: https://cloud.google.com/blog/products/ai-machine-learning/e...

This is a cutting edge subfield of ML, so it's understandable that one paper in a medical journal isn't going to be on that cutting edge, but I think they should at least acknowledge that their investigations barely scratched the surface.

Re: AI recognition of patient race in medical imaging: a modelling study

#95

Post by pdpi »

aulin wrote: post #18
tech-historian wrote: post #2

The interpretation part hit home: "The results from our study emphasise that the ability of AI deep learning models to predict self-reported race is itself not the issue of importance. However, our finding that AI can accurately predict self-reported race, even from corrupted, cropped, and noised medical images, often when clinical experts cannot, creates an enormous risk for all model deployments in medical imaging.…

what's this enormous risk they're talking about? racial bias in x-ray reading? race can be a risk factor in plenty of diseases, why should we actively try to remove this information from medical images?

ML models are great tools, but they're way too much of a black box. What you have here is a model that's predicting something you think it shouldn't have been possible to predict, and you can't simply ask it where that prediction comes from. Absent an explanation for how the model is doing this, you have to consider the possibility that whatever is poisoning that prediction will also poison others.

Re: AI recognition of patient race in medical imaging: a modelling study

#96

Post by MontyCarloHall »

KaiserPro wrote: post #65
hellohowareu wrote: post #5

Simply go to google image and search: "skeletal racial differences". subspecies are found across species-- they happen based on geographic dispersion and geographic isolation, which humans underwent for tens and hundreds of thousands of years. Welcome to the sciences of anatomy, anthropology, and forensics. other differences: - slow twitch vs fast twitch muscle - teeth shape - shapes and colors of various parts - gen…

> skeletal racial differences £10 says that its not that. Anatomy is extraordinarily hard, and AI isn't that good, yet. Sure different races have different layouts, but often that's only really obvious post mortem. (ie when you can yank out the bones and look at them, there are of course corner cases where high res CAT/MRI scans can pull out decent skeletal imagery in 3D) There are other cases, but that should be eas…

>I'd say its probably picking up on the style of imaging, rather than anything anatomical

Certainly possible! They do control for hospital and machine …

>Race prediction performance was also robust across models trained on single equipment and single hospital location on the chest x-ray and mammogram datasets

… but it’s also possible that different chest x-rays were being used for different diagnostic purposes and thus have a different imaging style, which a) may correlate with ethnicity and b) does not appear to be explicitly controlled for.

Re: AI recognition of patient race in medical imaging: a modelling study

#97

Post by SpicyLemonZest »

civilized wrote: post #94
SpicyLemonZest wrote: post #92

Earlier quoted context omitted.

For those of us less familiar with this space, what are these modern explanation tools? (I certainly agree that it's plausible the model is seeing a physiological difference, and the researchers seem to have considered a few concrete hypotheses on that dimension.)

Here's an introduction to one technique: https://cloud.google.com/blog/products/ai-machine-learning/e... This is a cutting edge subfield of ML, so it's understandable that one paper in a medical journal isn't going to be on that cutting edge, but I think they should at least acknowledge that their investigations barely scratched the surface.

Thanks! This looks pretty neat, I'll have to dig more into their explainable AI product later.

Re: AI recognition of patient race in medical imaging: a modelling study

#98

Post by ibejoeb »

matthewdgreen wrote: post #93
ibejoeb wrote: post #75

Earlier quoted context omitted.

Typically? It's coded in the standard. There's a DICOM tag for it. https://dicom.innolitics.com/ciods/procedure-log/patient/001...

Unlike the authors of this research paper I am not a trained clinician, so I can't tell you. However I would note that the first exemplary value in the link you gave me is "REMOVED".

It doesn't provide example data, but there's still a spot in the standard for it. The values can differ by modality or manufacturer. Sure, it's not required, but certainly it's very important in some situations. Consider dermoscopy.

If interested, searching for "dicom conformance" should yield lots of docs that probably contain specific values for those things.

Re: AI recognition of patient race in medical imaging: a modelling study

#99

Post by KaiserPro »

kerblang wrote: post #77

> Importantly, if used, such models would lead to more patients who are Black and female being *incorrectly* identified as healthy I think this is the point a lot of people are missing; they think, "So what if 'black' correlates to unhealthy and the model notices? It's just seeing the truth!" However, I'm still wondering how this incorrectness works; can anyone explain? Edit: Clue: The AI is predicting self-reported…

My guess is that they are using an american dataset. This I would suspect encodes socioeconomic data into the samples. ie rich people, have access to better diagnostics, get seen earlier and are treated sooner. Conversely poorer present later and with more obvious symptoms. also the type of system used to take the images would also be strongly correlated.

Re: AI recognition of patient race in medical imaging: a modelling study

#100

Post by chopin »

inglor_cz wrote: post #74
nerdponx wrote: post #31

Earlier quoted context omitted.

Then problem is that human experts sometimes can't tell the difference while the model can.

AI is also able to determine your sex from your retinal scan with very good levels of certainty (provided that your retina is healthy; its ability to tell sexes apart drops in diseased retinas). [0] Which came as a surprise to the ophthalmologists, because they aren't aware of any significant differences between male and female retinas. [0] https://www.researchgate.net/publication/351558516_Predictin...

I am surprised that this is a surprise. At least color vision is encoded in the X-Chromosome so there should be variation as males have only one which can be expressed.
Post reply on HN