Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

201–210 of 256 posts

Re: AI models miss disease in Black and female patients

#201

Post by ars »

protonbob wrote: post #49
mhuffman wrote: post #41

Earlier quoted context omitted.

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

I'm going to wager an uneducated guess. Black people are less likely to go to the doctor for both economic and historical reasons so images from them are going to be underrepresented. So in some way I guess you could say that yes, latent racism caused people to go to the doctor less which made them appear less in the data.

Men are also way less likely to go to Dr vs women. Yet this claims a bias against women as well.

Re: AI models miss disease in Black and female patients

#203

Post by genocidicbunny »

QuarterReptile wrote: post #198
genocidicbunny wrote: post #161

Earlier quoted context omitted.

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

>women tended to be underrepresented in western medical literature. Is there some evidence of this? It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to. It's more believable (still not very) that they disproportionately avoid studies.

There's a few factors here:

1. We're talking about a span of 200 or so years. There is plenty of modern medicine that is still based on now century+ old knowledge.

2. The feedback loop. If you were learning medicine in the 1950's, you were probably learning from medical texts written in the 50 or so years before that, when it's not unreasonable to think women would have been less represented. Those same doctors from the 1950's would then have been teaching the next generation of doctors, and they carried those (intentional or not) biases forward. Of course there was new information, but you don't tend to have much time to explore novel medicine when you're in medical school or residency, so by the time you can integrate the new knowledge, some biases have already set in. Repeat for a few generations, and you tend to only get a dilution of those old ideas, not a wholesale replacement of them.

3. If you've been affected by such biases as a patient, you're less likely to trust and be willing to participate with medicine, once more reinforcing the feedback loop.

I don't have any specific numbers or studies for you, but you could probably find more than a few that attest to this phenomenon. I hate to go with 'trust me bro' here, but my knowledge on this topic largely comes from knowing people that are either studying or practicing medicine currently, so it's anecdotal, but the anecdotes are from those in the field currently.

Re: AI models miss disease in Black and female patients

#204

Post by genocidicbunny »

klipt wrote: post #200
genocidicbunny wrote: post #191

Earlier quoted context omitted.

I think we're really talking about different aspects of the same issue. Everything you've described basically agrees with "those who have more access to medicine" because those are also the ones inherently more convenient to test/observe.

Like how the ones with the most access to medicine are mice, because they're convenient to experiment on.

And this is absolutely something one needs to consider when reading medical studies -- if they only use animal (usually mice) models, there's a decent chance the conclusions are not directly transferable to humans.

Re: AI models miss disease in Black and female patients

#205

Post by adamhartenz »

QuarterReptile wrote: post #198
genocidicbunny wrote: post #161

Earlier quoted context omitted.

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

>women tended to be underrepresented in western medical literature. Is there some evidence of this? It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to. It's more believable (still not very) that they disproportionately avoid studies.

The history of ADHD research is a common referenced example

Re: AI models miss disease in Black and female patients

#206

Post by lfmhd »

QuarterReptile wrote: post #198
genocidicbunny wrote: post #161

Earlier quoted context omitted.

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

>women tended to be underrepresented in western medical literature. Is there some evidence of this? It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to. It's more believable (still not very) that they disproportionately avoid studies.

> It's more believable (still not very) that they disproportionately avoid studies.

Women are definitely strongly underrepresented in medical texts, and it's not typically by choice: https://www.aamc.org/news/why-we-know-so-little-about-women-...

A lot of "the consensus" in medical literature predates the inclusion of women in medical research, and even still there things are not tested on women (often because of ethical risks around fertility and birth defects).

Re: AI models miss disease in Black and female patients

#207

Post by dragonwriter »

QuarterReptile wrote: post #198
genocidicbunny wrote: post #161

Earlier quoted context omitted.

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

>women tended to be underrepresented in western medical literature. Is there some evidence of this? It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to. It's more believable (still not very) that they disproportionately avoid studies.

> It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to

“Medical attention” and “coverage in medical literature” aren't even remotely the same thing, so dismissing a claim about the first based on your anecdotal experience of the second is completely bonkers.

Re: AI models miss disease in Black and female patients

#208

Post by stevenbedrick »

QuarterReptile wrote: post #198
genocidicbunny wrote: post #161

Earlier quoted context omitted.

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

>women tended to be underrepresented in western medical literature. Is there some evidence of this? It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to. It's more believable (still not very) that they disproportionately avoid studies.

There is indeed a lot of evidence of this but you've got the direction backwards- it's not that women avoid studies, it's that for a long time studies specifically excluded women. Ditto for people of different races. This is why these days (well, as of today, at least) the NIH has a whole set of very well-established policies around inclusion in clinical trials that include sex, race, and age: https://grants.nih.gov/policy-and-compliance/policy-topics/i...

And this isn't for "DEI" reasons, it's literally because for decades there used to be drug trials that excluded women and as a result ended up releasing drugs that gave half the population weird side effects that didn't get caught during the trials, or just plain didn't work as well on one group or another in ways that were really hard to debug once the drug was on the market. That was legit bad science, and the medical research world has worked very hard over the last thirty years to do better. We are admittedly not there yet, but things are a lot better than they used to be.

For a really interesting take on the history of racial exclusion and bias in medicine, I recommend Uché Blackstock's recent book "Legacy: A Black Physician Reckons With Racism In Medicine" which gave a great overview.

Oh! And also everybody should read Abby Norman's "Ask Me About My Uterus," it gives a fabulous history of issues around women's health.

Re: AI models miss disease in Black and female patients

#209

Post by dartos »

prasadjoglekar wrote: post #173
dartos wrote: post #168

Earlier quoted context omitted.

> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women. The biases and mistakes of those who created the data set leak into the model. Early image recognition models had some very… culturally…

Xays by definition don't look at skin color. Do chest x-rays of black women reveal that there's something different about their chests than white or asian women? That doesn't pass my non doctor sniff test, but someone can correct me (no sarcasm intended).

There can be differences which statistical models pick up which we humans don’t.

For example, a couple years ago there was a statistical model made which could fairly accurately predict (iirc >80%) the gender of a person based on a picture of their iris. At the time we didn’t know there was a visible iris difference between genders, but a statistical model found one.

That’s kind of the whole point of statistical classification models. Feed in a ton of data and the model will discover the differentiating features.

Put another way, If we knew all the possible differences between someone with cancer and without, we wouldn’t need statistical models at all, we could just automate the diagnosis.

We don’t know the indicators that we don’t know, so we don’t know if some possible indicators show up or don’t show up in a given group of people.

That is the danger of wholly relying on statistical models.

Re: AI models miss disease in Black and female patients

#210

Post by timewizard »

itishappy wrote: post #133
timewizard wrote: post #86

Earlier quoted context omitted.

LLMs don't and cannot want things. Human beings also like it when the future is mostly like the past. They just call that "predictability." Human data is bias. You literally cannot remove one from the other. There are some people who want to erase humanity's will and replace it with an anthropomorphized algorithm. These people concern me.

Can humans want things? Our reward structures sure seem aligned in a manner that encourages anthropomorphization. Biases are symptoms of imperfect data, but that's hardly a human-specific problem.

> Can humans want things?

Yes. Do I have to prompt you? Or do you exist on your own?

> Our reward structures sure seem aligned in a manner that encourages anthropomorphization.

You do understand what that word /means/?

> are symptoms of imperfect data

Which means humans cannot generate perfect data. So good luck with all that high priced "training" you're doing. Mathematically errors compound.

Post reply on HN