Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI recognition of patient race in medical imaging: a modelling study

thelancet.com

Post reply on HN
Print view View original

101–110 of 180 posts

Re: AI recognition of patient race in medical imaging: a modelling study

#101

Post by PartiallyTyped »

dijit wrote: post #38
PartiallyTyped wrote: post #20

Earlier quoted context omitted.

One of the reasons certain communities were hit harder with Covid was vit D deficiency as a consequence of skin color.

That is one hypothesised cause for the disparity, social factors in those cases need to be controlled for. A better discussion is around sickle cell anaemia[0] which is exclusively carried by people of African or Afro-Caribbean descent. [0]: https://en.wikipedia.org/wiki/Sickle_cell_disease

That's a better example, thank you. Reminded me that I know quite a few people with Thalassemia/Mediterranean anemia.

Re: AI recognition of patient race in medical imaging: a modelling study

#102

Post by Loughla »

codefreeordie wrote: post #71
aulin wrote: post #58

Earlier quoted context omitted.

ok, maybe it's an US specific thing, why wouldn't a clinical radiologist have all the information he can gather about his patient including race to help the diagnosis?

Because in the US we are required to pretend that there is no such thing as race and no such thing as gender, and all people are exactly and precisely the same and there can be no differences.

Not to get into a flame war, but I want to present an alternate option to yours.

Because in the US some people have a hard time understanding that all races and genders deserve to be treated equally as humans with the same access to goods and services. Further, that there are disparities in care based on race/ethnicity[1][2] and gender[3][4] because of that racism/sexism present in the systems. This then leads to requiring that race/ethnicity and gender data be scrubbed sometimes to keep people from impacting outcomes based on their own biases.

[1] https://www.americanbar.org/groups/crsj/publications/human_r...

[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1924616/

[3] https://www.americashealthrankings.org/learn/reports/2019-se...

[4] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2965695/

Re: AI recognition of patient race in medical imaging: a modelling study

#103

Post by civilized »

jl6 wrote: post #30

> Models trained on low-pass filtered images maintained high performance even for highly degraded images. More strikingly, models that were trained on high-pass filtered images maintained performance well beyond the point that the degraded images contained no recognisable structures; to the human coauthors and radiologists it was not clear that the image was an x-ray at all. What voodoo have they unearthed?

I think there's a significantly greater than zero chance that they simply botched their ML pipeline horribly and would get their 0.98 AUCs from completely blank images.

Re: AI recognition of patient race in medical imaging: a modelling study

#104

Post by dragonwriter »

proto-n wrote: post #51
jl6 wrote: post #30

> Models trained on low-pass filtered images maintained high performance even for highly degraded images. More strikingly, models that were trained on high-pass filtered images maintained performance well beyond the point that the degraded images contained no recognisable structures; to the human coauthors and radiologists it was not clear that the image was an x-ray at all. What voodoo have they unearthed?

I tend to not believe unbelievable results in machine learning. It's too easy to unintenionally cause some kind of information leakage. I haven't read the paper in detail though, so their experimentation setup could be foolproof, this is not a critique of this paper specifically.

> I tend to not believe unbelievable results

That seems tautologically true.

Re: AI recognition of patient race in medical imaging: a modelling study

#105

Post by amarshall »

codefreeordie wrote: post #76
amarshall wrote: post #54

Earlier quoted context omitted.

Just because the model relies on race in some way doesn’t mean that we know it relies on it. I.e., the model is, unbeknownst to us, biased on race in inaccurate ways.

Presumably the model would actually be biased on race in accurate ways, if it found the correlation itself

Maybe, maybe not. Hard to say—which is the problem they call out in the paper

> efforts to control [model race-prediction] when it is undesirable will be challenging and demand further study

Re: AI recognition of patient race in medical imaging: a modelling study

#106

Post by codefreeordie »

Loughla wrote: post #102
codefreeordie wrote: post #71

Earlier quoted context omitted.

Because in the US we are required to pretend that there is no such thing as race and no such thing as gender, and all people are exactly and precisely the same and there can be no differences.

Not to get into a flame war, but I want to present an alternate option to yours. Because in the US some people have a hard time understanding that all races and genders deserve to be treated equally as humans with the same access to goods and services. Further, that there are disparities in care based on race/ethnicity[1][2] and gender[3][4] because of that racism/sexism present in the systems. This then leads to req…

It sometimes makes sense to scrub race/ethnicity/gender information from certain types of data, typically when a human is going to be making individual decisions.

For example, not having race data on resumes is generally productive, because that categorization can't provide a meaningful input to the decision associated with an individual person. Even if it were to be the case that there was some correlation between race and skill at whatever job you're interviewing for[1], the size of the effect is almost certainly small, and in the meanwhile you've also controlled for any bias in the person doing the reviewing.

If you're having a machine look at a dataset, and the machine determines that race or ethnicity is a material factor in determining some attribute in that dataset, you're not doing anybody any good by denying that fact and destroying the result.

[1]Let's ignore for the purposes of this discussion, fields (like certain sports) where extreme competition combines with a position heavily dependent upon racially-linked physical characteristics. Though even in this case, there is still a (different, weaker) argument for suppressing race data in "resumes" (yes, I know, ballplayers don't submit resumes to their local NBA franchise)

Re: AI recognition of patient race in medical imaging: a modelling study

#107

Post by codefreeordie »

amarshall wrote: post #105
codefreeordie wrote: post #76

Earlier quoted context omitted.

Presumably the model would actually be biased on race in accurate ways, if it found the correlation itself

Maybe, maybe not. Hard to say—which is the problem they call out in the paper > efforts to control [model race-prediction] when it is undesirable will be challenging and demand further study

The correlation being "undesirable" to the individuals doing the research does not mean that the correlation is inaccurate.

I mean, sure, there are tons of ways for garbage data to sneak into ML models -- though these guys tried pretty hard to control for that -- but if the model actually determined that "race" is a meaningful feature, then that might be because it is, and science should be concerned with what is, not with what we wish were.

Re: AI recognition of patient race in medical imaging: a modelling study

#108

Post by krona »

tedivm wrote: post #90
KaiserPro wrote: post #65

Earlier quoted context omitted.

> skeletal racial differences £10 says that its not that. Anatomy is extraordinarily hard, and AI isn't that good, yet. Sure different races have different layouts, but often that's only really obvious post mortem. (ie when you can yank out the bones and look at them, there are of course corner cases where high res CAT/MRI scans can pull out decent skeletal imagery in 3D) There are other cases, but that should be eas…

>If I had to bet, and I knew where the data was coming from, I'd say its probably picking up on the style of imaging, rather than anything anatomical. Not all x-rays have bones in, and not all bones differ reliably to detect race. This was my guess as well. I've spent a lot of time around radiology and AI (I used to work at a company specializing in it) and we read a lot of the failure cases as well. There was one ex…

> one hospital was for higher risk patients- so it learned to assign all patients from that hospital to the disease category simply because they were at that hospital.

That just sounds like poor feature selection/engineering. Garbage in, garbage out.

Re: AI recognition of patient race in medical imaging: a modelling study

#109

Post by cameldrv »

towaway15463 wrote: post #70
matthewdgreen wrote: post #42

Earlier quoted context omitted.

"This issue creates an enormous risk for all model deployments in medical imaging: if an AI model relies on its ability to detect racial identity to make medical decisions, but in doing so produced race-specific errors, clinical radiologists (who do not typically have access to racial demographic information) would not be able to tell, thereby possibly leading to errors in health-care decision processes."

Without knowing the actual outcome, isn’t there also a possibility of error due to not knowing the race of the individual? They used mammogram images in the study and it is well known that incidence of breast cancer varies by race. Removing that information from the model could result in worse performance.

Well one thing you wouldn’t want to do is take the output of this model and then apply a correction factor for race on top of it, because the model is already taking that into account.

Re: AI recognition of patient race in medical imaging: a modelling study

#110

Post by inglor_cz »

chopin wrote: post #100
inglor_cz wrote: post #74

Earlier quoted context omitted.

AI is also able to determine your sex from your retinal scan with very good levels of certainty (provided that your retina is healthy; its ability to tell sexes apart drops in diseased retinas). [0] Which came as a surprise to the ophthalmologists, because they aren't aware of any significant differences between male and female retinas. [0] https://www.researchgate.net/publication/351558516_Predictin...

I am surprised that this is a surprise. At least color vision is encoded in the X-Chromosome so there should be variation as males have only one which can be expressed.

It is a surprise, because the retina as an organ is very well visible and observable in living people, so we have a ton of observational data and practical clinical experience. But despite that, humans haven't noticed anything.
Post reply on HN