Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

51–60 of 256 posts

Re: AI models miss disease in Black and female patients

#51

Post by sergiotapia »

Animats wrote: post #8

What's so striking is how strongly race shows in X-rays. That's unexpected.

It's odd how we can segment between different species in animals, but in humans it's taboo to talk about this. Threw the baby out with the baby water. I hope we can fix this soon so everybody can benefit from AI. The fact that I'm a male latino should be an input for an AI trained on male latinos! I want great care!

I don't want pretend kumbaya that we are all humans in the end. That's not true. We are distinct! We all deserve love and respect and care, but we are distinct!

Re: AI models miss disease in Black and female patients

#52

Post by Avshalom »

nonethewiser wrote: post #29
nradov wrote: post #22

Earlier quoted context omitted.

False positive diagnoses cause a huge amount of patient harm. New technologies should only be deployed on a widespread basis when they are justified based on solid evidence-based medicine criteria.

No one says you have to use the AI models stupidly. If it works poorly for black women and female women dont use it for them. Or simply dont use it for the initial diagnosis. Use it after the normal diagnosis process as more of a validation step. Anyways, this all points to the need to capture biological information as input or even having seperately models tuned to different factors.

Every single AI company says you should use AI models stupidly. Replacing experts is the whole selling point.

Re: AI models miss disease in Black and female patients

#53

Post by kjkjadksj »

This isn’t an AI problem but a general medical field problem. It is a big issue with basically any population centric analysis where the people involved in the study don’t have a perfect subset of the worlds population to model human health; they have a couple hundred blood samples from patients at a Boise hospital over the past 10 years perhaps. And they validate this population against some other available cohort that is similarly constrained by what is practically possible to sample and catalog and might not even see the same markers shake out between disease and healthy.

There are a couple populations that are really overrepresented as a result of these available datasets. Utah populations on one hand because they are genetically bottlenecked and therefore have better signal to noise in theory. And on the other the Yoruba tribe out of west africa as a model of the most diverse and ancestral population of humans for studies that concern themselves with how populations evolved perhaps.

There are other projects too amassing population data. About 2/3rd of the population of iceland has been sequenced and this dataset is also frequently used.

Re: AI models miss disease in Black and female patients

#54

Post by nonethewiser »

lalaithion wrote: post #40
orand wrote: post #35

Earlier quoted context omitted.

Race and sex should be inputs. Giving any medical prominence to gender identity will result in people receiving wrong and potentially harmful treatment, or lack of treatment.

Most trans people have undergone gender affirming medical care. A trans man who has had a hysterectomy and is on testosterone will have a very different medical baseline than a cis woman. A trans woman who has had an orchiectomy and is on estrogen will have a very different medical baseline than a cis man. It is literally throwing out relevant medical information to attempt to ignore this.

How is that in any way in conflict with what he said? You're just making an argument for more inputs.

Biological sex, hormone levels, etc.

Re: AI models miss disease in Black and female patients

#57

Post by acobster »

nonethewiser wrote: post #29
nradov wrote: post #22

Earlier quoted context omitted.

False positive diagnoses cause a huge amount of patient harm. New technologies should only be deployed on a widespread basis when they are justified based on solid evidence-based medicine criteria.

No one says you have to use the AI models stupidly. If it works poorly for black women and female women dont use it for them. Or simply dont use it for the initial diagnosis. Use it after the normal diagnosis process as more of a validation step. Anyways, this all points to the need to capture biological information as input or even having seperately models tuned to different factors.

> having seperately models tuned to different factors.

Sure. Separate but equal, presumably.

Re: AI models miss disease in Black and female patients

#58

Post by ideamotor »

orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

I really can’t help but think of the simulation hypothesis. What are the chances this copy-cat technology was developed when I was alive, given that it keeps going.

Re: AI models miss disease in Black and female patients

#59

Post by daveguy »

mhuffman wrote: post #41
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

You really just have to understand one thing: AI is not intelligent. It's pattern matching without wisdom. If fewer people in the dataset are a particular race or gender it will do a shittier job predicting and won't even "understand" why or that it has bias, because it doesn't understand anything at a human level or even a dog level. At least most humans can learn their biases.
Post reply on HN