Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

41–50 of 256 posts

Re: AI models miss disease in Black and female patients

#41

Post by mhuffman »

orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. "

... very interesting that the inputs to the model had nothing related to race or gender, but somehow it still was able to miss diagnose Black and female patients? I am curious of the mechanism for this. Can it just tell which x-rays belong to Black or female patients and then use some latent racism or misogyny to change the diagnosis? I do remember when it came out that AI could predict race from medical images with no other information[1], so that part seems possible. But where would it get the idea to do a worse diagnosis, even if it determines this? Surely there is no medical literature that recommends this!

[1]https://news.mit.edu/2022/artificial-intelligence-predicts-p...

Re: AI models miss disease in Black and female patients

#42

Post by onlyrealcuzzo »

elietoubi wrote: post #14

I came across a fascinating Microsoft research paper on MedFuzz ( https://www.microsoft.com/en-us/research/blog/medfuzz-explor... ) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaund…

It's almost as if you'd want to not feed what the patient says directly to an LLM.

A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record.

You'd want to have a charting stage before you send the patient input to the LLM.

It's probably not important whether the patient is low income or high income or whether they live in the hood or the uppity part of town.

Re: AI models miss disease in Black and female patients

#43

Post by cheschire »

elietoubi wrote: post #14

I came across a fascinating Microsoft research paper on MedFuzz ( https://www.microsoft.com/en-us/research/blog/medfuzz-explor... ) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaund…

Can't the same be said for humans though? Not to be too reductive, but aren't most general practitioners just pattern recognition machines?

Re: AI models miss disease in Black and female patients

#44

Post by rvnx »

megaman821 wrote: post #33
Larrikin wrote: post #10

Earlier quoted context omitted.

[flagged]

If I'm a black female in a remote village with no doctor, I'm absolutely going to roll the dice with AI. In the mean time these problems should be surfaced and corrected.

Even if you are not a remote black woman:

AI can be a serious helping hand when trying to figure out what diseases you have.

Not everybody has access to doctors, sometimes, whole countries don't have good specialists in everything.

When Google refuses to answer medical questions (when it could have answered and said "be careful, I'm not really sure"), it's actually putting lives in danger by hiding useful information.

If you can, of course it's better to go to a triple Nobel Prize doctor that is 200 USD per session, but not everybody can, and these doctors may not even be available in your country.

But otherwise in general, it's better to have the opinion of AI than nothing.

Re: AI models miss disease in Black and female patients

#45

Post by nradov »

nonethewiser wrote: post #29
nradov wrote: post #22

Earlier quoted context omitted.

False positive diagnoses cause a huge amount of patient harm. New technologies should only be deployed on a widespread basis when they are justified based on solid evidence-based medicine criteria.

No one says you have to use the AI models stupidly. If it works poorly for black women and female women dont use it for them. Or simply dont use it for the initial diagnosis. Use it after the normal diagnosis process as more of a validation step. Anyways, this all points to the need to capture biological information as input or even having seperately models tuned to different factors.

The guidelines on how to use a particular AI model can only be written after extensive clinical research and data analysis. You can't skip that step without endangering patients, and it will take years to do properly for each one.

Re: AI models miss disease in Black and female patients

#46

Post by tawayf1 »

_bin_ wrote: post #3

This seems like a problem that should be worked on It also seems like we shouldn't let it prevent all AI deployment in the interim. It is better that we take the disease detection rate for part of the population up a few percent than we do not. Plus it's not like doctors or radiologists always diagnose at perfectly equal accuracy across all populations. Let's not let the perfect become the enemy of the good.

[flagged]

Re: AI models miss disease in Black and female patients

#47

Post by nonethewiser »

andsoitis wrote: post #31
nonethewiser wrote: post #19

Race and gender should be inputs then. The female part is actually a bit more surprising. Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. But female? Thats ~52% globally.

> Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. What about Africa?

The story is that there exists this model which poorly predicts for black (and female) patients. Given there are probably lots of datasets where black people are a vast minority makes this not surprising.

For all I know there are millions of models with extremely poor accuracy based on African datasets. Wouldnt really change anything about the above though. I wouldnt expect that though and it would definitely be interesting.

Re: AI models miss disease in Black and female patients

#48

Post by acobster »

yieldcrv wrote: post #9

just giving globs of training sets and letting a process cook for a few months is just going to be seen as lazy in the near future more specialization of models is necessary, now that there is awareness

Specialization in what though? Do you really think VCs are going to drive innovation on equitable outcomes? Where is the money in that? I have a hunch that oppression will continue to be profitable.

Re: AI models miss disease in Black and female patients

#49

Post by protonbob »

mhuffman wrote: post #41
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

I'm going to wager an uneducated guess. Black people are less likely to go to the doctor for both economic and historical reasons so images from them are going to be underrepresented. So in some way I guess you could say that yes, latent racism caused people to go to the doctor less which made them appear less in the data.
Post reply on HN