Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

191–200 of 256 posts

Re: AI models miss disease in Black and female patients

#191

Post by genocidicbunny »

belorn wrote: post #189
genocidicbunny wrote: post #161

Earlier quoted context omitted.

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

It is very true that a lot of medical knowledge is gained empirically, and there is also an additional aspect to it. The history of Medical research is generally studied on the demographics where such testing is cultural acceptable, and where the gains of such research has been mostly sought, which is young men drafted into wars. The second common demographic are medical students, which historically was biased toward…

I think we're really talking about different aspects of the same issue. Everything you've described basically agrees with "those who have more access to medicine" because those are also the ones inherently more convenient to test/observe.

Re: AI models miss disease in Black and female patients

#192

Post by ironSkillet »

darth_avocado wrote: post #187
dartos wrote: post #168

Earlier quoted context omitted.

> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women. The biases and mistakes of those who created the data set leak into the model. Early image recognition models had some very… culturally…

I am confused. I’m not a doctor, but why would a model perform poorly at detecting diseases in X-rays in different genders and races unless the diseases present themselves differently in X-Rays for different races? Shouldn’t the model not have the race and gender information to begin with? Like a model trained on detecting lesions should perform equally well on ANY X-Ray unless lesions show up differently in differen…

You and the article are both correct. The disease does present itself differently as a function of these other characteristics, so since the training dataset doesn't contain enough samples of these different presentations, it is unable to effectively diagnose.

Re: AI models miss disease in Black and female patients

#193

Post by intuitionist »

KittenInABox wrote: post #113
dekhn wrote: post #108

Earlier quoted context omitted.

It doesn't seem surprising at all. Genetic history correlates with race, and genetic history correlates with body-level phenotypes; race also correlates with socioeconomic status which correlates with body-level phenotypes. They are of course fairly complex correlations with many confounding factors and uncontrolled variables. It has been controversial to discuss this and a lot of discussions about this end up in fla…

What is the body-level phenotype of a ribcage by race? I think what baffles me is that black people as a group are more genetically diverse than every other race put together so I have no idea how you would identify race by ribcage x-rays exclusively.

Sub-Saharan Africans are extremely genetically diverse but a sample of ~100 Black Americans is unlikely to have any Khoekhoe or Twa representation.

Anyway it’s possible that the model can pick up on other cues as well; if you had some X-rays from a hospital in Portland, Oregon and some from a hospital in Montgomery, Alabama and some quirk of the machine in Montgomery left artifacts that a model could pick up on, the presence of those artifacts would be quite correlated with race.

Re: AI models miss disease in Black and female patients

#194

Post by genocidicbunny »

prasadjoglekar wrote: post #173
dartos wrote: post #168

Earlier quoted context omitted.

> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women. The biases and mistakes of those who created the data set leak into the model. Early image recognition models had some very… culturally…

Xays by definition don't look at skin color. Do chest x-rays of black women reveal that there's something different about their chests than white or asian women? That doesn't pass my non doctor sniff test, but someone can correct me (no sarcasm intended).

But they do look at bones and near-bone tissues, which can still have variance based on ethnicity and gender. For a really brute-force example, just think about how we use the shape of the pelvis and some other bones to identify the gender of skeletal remains of a person. If you had a data set of pelvic xrays that only included males, your data set would imply that female pelvic bones are massively malformed despite being perfectly normal for that gender.

Re: AI models miss disease in Black and female patients

#195

Post by ineedaj0b »

sc68cal wrote: post #184
prasadjoglekar wrote: post #173

Earlier quoted context omitted.

Xays by definition don't look at skin color. Do chest x-rays of black women reveal that there's something different about their chests than white or asian women? That doesn't pass my non doctor sniff test, but someone can correct me (no sarcasm intended).

What groups have the financial means to get chest x-rays, and what groups do not? What historical events could create the circumstances where different groups have different health outcomes?

you ain't gonna like the truth but there are differences between the races and during med school they try to say it ain't so but once you start seeing patients there's differences in musculature/skin, all sorts. and if you have a good attending they tactfully tell you and you go 'was it in a study?' and nope nobody wants to publish it. and no i'm talking just stuff like scabies or diabetes.

Re: AI models miss disease in Black and female patients

#196

Post by LadyCailin »

nonethewiser wrote: post #74
LadyCailin wrote: post #65

Earlier quoted context omitted.

That’s mostly correct, that “gender identity” doesn’t matter for physical medicine. But hormone levels and actual internal organ sets matter a huge amount, more than genes or original genitalia, in general. There are of course genetically linked diseases, but there are people with XX chromosomes that are born with a penis, and XY people that are born with a vulva, and genetically linked diseases don’t care about exte…

>But hormone levels and actual internal organ sets matter a huge amount, more than genes or original genitalia Or current genitalia for that matter. It's just a matter of the genitalia signifying other biological realities for 99.9% of people. For sure more info like average hormone levels or ranges over time would be more helpful.

Yeah, sure, and for most people it’s a fair enough proxy. But if it has to be boiled down to exactly one of “M” or “F”, then “birth sex” must not be the deciding factor. If it must be a single criteria, it should be current hormone levels, artificial or not. And, since most trans people who actually transition and live as their preferred gender identity are on hormones, “gender identity” is a good proxy for 99.99% of the population, including the set of people for who “birth genitalia” is also a good proxy. But ideally, it doesn’t get simplified this much in the first place. And of course, it doesn’t, in practice, because most people actually form a relationship with their doctor, and they treat holistically, based on individual factors, and not simply whether the medical record says M or F.

But, if we must over generalize, “gender identity” really is the most useful proxy, in fact, and it also happily happens to be quite inclusive too.

Of course this conversation started from a transphobic viewpoint, which doesn’t actually care about any of these distinctions anyways, regardless of the merit, it’s just someone being triggered about someone respecting someone else’s gender identity.

Re: AI models miss disease in Black and female patients

#197

Post by gonzobonzo »

belorn wrote: post #189
genocidicbunny wrote: post #161

Earlier quoted context omitted.

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

It is very true that a lot of medical knowledge is gained empirically, and there is also an additional aspect to it. The history of Medical research is generally studied on the demographics where such testing is cultural acceptable, and where the gains of such research has been mostly sought, which is young men drafted into wars. The second common demographic are medical students, which historically was biased toward…

> The history of Medical research is generally studied on the demographics where such testing is cultural acceptable, and where the gains of such research has been mostly sought, which is young men drafted into wars.

Though in this study, the AI models were also biased against people under the age of 40.

It is interesting that we're also seeing a lot of bias in the reporting and discussion of these results. The results tested three groups for bias, and found a bias in all three. Yet the headline only mentions the bias against two of the groups, and almost the entirety of the discussion here only talks about bias against two of the groups while ignoring the third group.

If I test a system for bias, select three different groups to test for, and all three have a bias against them, my first reaction would be "there's a good chance that it's also biased against many other groups, I should test for those as well." It wouldn't be to pretend that there's only bias against the only three groups I actually bothered checking for. It definitely wouldn't be two ignore one of those groups, and pretend that there's only a bias against the other two.

Re: AI models miss disease in Black and female patients

#198

Post by QuarterReptile »

genocidicbunny wrote: post #161
MountainArras wrote: post #90

Earlier quoted context omitted.

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

>women tended to be underrepresented in western medical literature.

Is there some evidence of this? It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to. It's more believable (still not very) that they disproportionately avoid studies.

Re: AI models miss disease in Black and female patients

#199

Post by QuarterReptile »

prasadjoglekar wrote: post #173
dartos wrote: post #168

Earlier quoted context omitted.

> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women. The biases and mistakes of those who created the data set leak into the model. Early image recognition models had some very… culturally…

Xays by definition don't look at skin color. Do chest x-rays of black women reveal that there's something different about their chests than white or asian women? That doesn't pass my non doctor sniff test, but someone can correct me (no sarcasm intended).

Yes.

Re: AI models miss disease in Black and female patients

#200

Post by klipt »

genocidicbunny wrote: post #191
belorn wrote: post #189

Earlier quoted context omitted.

It is very true that a lot of medical knowledge is gained empirically, and there is also an additional aspect to it. The history of Medical research is generally studied on the demographics where such testing is cultural acceptable, and where the gains of such research has been mostly sought, which is young men drafted into wars. The second common demographic are medical students, which historically was biased toward…

I think we're really talking about different aspects of the same issue. Everything you've described basically agrees with "those who have more access to medicine" because those are also the ones inherently more convenient to test/observe.

Like how the ones with the most access to medicine are mice, because they're convenient to experiment on.
Post reply on HN