Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

61–70 of 256 posts

Re: AI models miss disease in Black and female patients

#61

Post by schnable »

sergiotapia wrote: post #51
Animats wrote: post #8

What's so striking is how strongly race shows in X-rays. That's unexpected.

It's odd how we can segment between different species in animals, but in humans it's taboo to talk about this. Threw the baby out with the baby water. I hope we can fix this soon so everybody can benefit from AI. The fact that I'm a male latino should be an input for an AI trained on male latinos! I want great care! I don't want pretend kumbaya that we are all humans in the end. That's not true. We are distinct! We a…

That's because humans are all the same species.

Re: AI models miss disease in Black and female patients

#62

Post by nradov »

onlyrealcuzzo wrote: post #42
elietoubi wrote: post #14

I came across a fascinating Microsoft research paper on MedFuzz ( https://www.microsoft.com/en-us/research/blog/medfuzz-explor... ) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaund…

It's almost as if you'd want to not feed what the patient says directly to an LLM. A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record. You'd want to have a charting stage before you send the patient input to the LLM. It's probably not important whethe…

I generally agree, however socioeconomic and environmental factors are highly correlated with certain medical conditions (social determinants of health). In some cases even causative. For example, patients who live near an oil refinery are more likely to have certain cancers or lung diseases.

https://doi.org/10.1093/jncics/pkaa088

Re: AI models miss disease in Black and female patients

#63

Post by nonethewiser »

NotYourLawyer wrote: post #56

[flagged]

Interesting catch. TIL I learned about this split.

>At the Columbia Journalism Review, we capitalize Black, and not white, when referring to groups in racial, ethnic, or cultural terms. For many people, Black reflects a shared sense of identity and community. White carries a different set of meanings; capitalizing the word in this context risks following the lead of white supremacists.

https://www.cjr.org/analysis/capital-b-black-styleguide.php

Re: AI models miss disease in Black and female patients

#65

Post by LadyCailin »

orand wrote: post #35
nonethewiser wrote: post #19

Race and gender should be inputs then. The female part is actually a bit more surprising. Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. But female? Thats ~52% globally.

Race and sex should be inputs. Giving any medical prominence to gender identity will result in people receiving wrong and potentially harmful treatment, or lack of treatment.

That’s mostly correct, that “gender identity” doesn’t matter for physical medicine. But hormone levels and actual internal organ sets matter a huge amount, more than genes or original genitalia, in general. There are of course genetically linked diseases, but there are people with XX chromosomes that are born with a penis, and XY people that are born with a vulva, and genetically linked diseases don’t care about external genitalia either way.

You simply can’t reduce it to birth sex assignment and that’s it, if you do, you will, as you say, end up with wrong and potentially harmful treatment, or lack of treatment.

Re: AI models miss disease in Black and female patients

#66

Post by appleorchard46 »

Freak_NL wrote: post #30
nonethewiser wrote: post #19

Race and gender should be inputs then. The female part is actually a bit more surprising. Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. But female? Thats ~52% globally.

Surprising? That's not a new realisation. It's a well known fact that women are affected by this in medicine. You can do a cursory search for the gender gap in medicine and get an endless amount of reporting on that topic.

I learned about this recently! It's wild how big the difference is. Even though legal/practical barriers to gender equality in medicine and data collection have been virtually nonexistent for the past few decades the inertia from the decades before that (where women were often specifically excluded, among many other factors) still weigh heavily.

To any women who happen to be reading this: if you can, please help fix this! Participate in studies, share your data when appropriate. If you see how a process can be improved to be more inclusive then please let it be known. Any (reasonable) male knows this is an issue and wants to see it fixed but it's not clear what should be done.

Re: AI models miss disease in Black and female patients

#67

Post by bko »

orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

Suppose you have a system that saves 90% of lives on group A but only 80% of lives in group B.

This is due to the fact that you have considerably more training data on group A.

You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A.

So the obvious thing to do is to have the technology intentionally kill ~1 out of every 10 patients from group A so the efficacy rate is ~80% for both groups. Problem solved

From the article:

> “What is clear is that it’s going to be really difficult to mitigate these biases,” says Judy Gichoya, an interventional radiologist and informatician at Emory University who was not involved in the study. Instead, she advocates for smaller, but more diverse data sets that test these AI models to identify their flaws and correct them on a small scale first. Even so, “Humans have to be in the loop,” she says. “AI can’t be left on its own.”

Quiz: What impact would smaller data sets have on efficacy for group A? How about group B? Explain your reasoning

Re: AI models miss disease in Black and female patients

#68

Post by yieldcrv »

acobster wrote: post #48
yieldcrv wrote: post #9

just giving globs of training sets and letting a process cook for a few months is just going to be seen as lazy in the near future more specialization of models is necessary, now that there is awareness

Specialization in what though? Do you really think VCs are going to drive innovation on equitable outcomes? Where is the money in that? I have a hunch that oppression will continue to be profitable.

the model involved in this article was developed by Stanford, and tested by UCLA

so yes I do believe that models will be created with more specific datasets, which is the specialization I was referring to

Re: AI models miss disease in Black and female patients

#69

Post by Avshalom »

krapp wrote: post #27
nonethewiser wrote: post #19

Race and gender should be inputs then. The female part is actually a bit more surprising. Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. But female? Thats ~52% globally.

Modern medicine has long operated under the assumption that whatever makes sense in a male body also makes sense in a female body, and womens' health concerns were often dismissed, misdiagnosed or misunderstood in patriarchal society. Women were rarely even included in medical trials prior to 1993. As a result, there is simply a dearth of medical research directly relevant to women for models to even train on.

https://www.npr.org/2022/11/01/1133375223/the-first-female-c... Twenty Twenty Two!

Re: AI models miss disease in Black and female patients

#70

Post by onlyrealcuzzo »

nradov wrote: post #62
onlyrealcuzzo wrote: post #42

Earlier quoted context omitted.

It's almost as if you'd want to not feed what the patient says directly to an LLM. A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record. You'd want to have a charting stage before you send the patient input to the LLM. It's probably not important whethe…

I generally agree, however socioeconomic and environmental factors are highly correlated with certain medical conditions (social determinants of health). In some cases even causative. For example, patients who live near an oil refinery are more likely to have certain cancers or lung diseases. https://doi.org/10.1093/jncics/pkaa088

So that's the important part, not that they're low income.
Post reply on HN