Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

111–120 of 256 posts

Re: AI models miss disease in Black and female patients

#111

Post by bko »

MountainArras wrote: post #90
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

Apparently providing this messy rough categorization appeared to help in some cases. From the article:

> To force CheXzero to avoid shortcuts and therefore try to mitigate this bias, the team repeated the experiment but deliberately gave the race, sex, or age of patients to the model together with the images. The model’s rate of “missed” diagnoses decreased by half—but only for some conditions.

In the end though I think you're right and we're just at the phases of hand-coding attributes. The bitter lesson always prevails

https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...

Re: AI models miss disease in Black and female patients

#113

Post by KittenInABox »

dekhn wrote: post #108
Animats wrote: post #8

What's so striking is how strongly race shows in X-rays. That's unexpected.

It doesn't seem surprising at all. Genetic history correlates with race, and genetic history correlates with body-level phenotypes; race also correlates with socioeconomic status which correlates with body-level phenotypes. They are of course fairly complex correlations with many confounding factors and uncontrolled variables. It has been controversial to discuss this and a lot of discussions about this end up in fla…

What is the body-level phenotype of a ribcage by race?

I think what baffles me is that black people as a group are more genetically diverse than every other race put together so I have no idea how you would identify race by ribcage x-rays exclusively.

Re: AI models miss disease in Black and female patients

#114

Post by milesrout »

potsandpans wrote: post #91
bko wrote: post #67

Earlier quoted context omitted.

Suppose you have a system that saves 90% of lives on group A but only 80% of lives in group B. This is due to the fact that you have considerably more training data on group A. You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. So the obvious thing to do is to have the technology intentionally kill ~1 out of every 10 patients from group A so the efficacy…

Imagine if you had a strawman so full of straw, it was the most strawfilled man that ever existed.

It is a hypothetical example not a strawman.

Re: AI models miss disease in Black and female patients

#115

Post by ericmcer »

pelorat wrote: post #107
MountainArras wrote: post #90

Earlier quoted context omitted.

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

I think the model needs to be thought about human anatomy, not just fed a bunch of scans. It needs to understand what ribs and organs are.

I don't think LLMs can achieve "understanding" in that sense.

Re: AI models miss disease in Black and female patients

#116

Post by airstrike »

dap wrote: post #72
onlyrealcuzzo wrote: post #42

Earlier quoted context omitted.

It's almost as if you'd want to not feed what the patient says directly to an LLM. A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record. You'd want to have a charting stage before you send the patient input to the LLM. It's probably not important whethe…

> It's almost as if you'd want to not feed what the patient says directly to an LLM. > A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record. I think the hard part of medicine -- the part that requires years of school and more years of practical experien…

By the way, the show The Pitt currently on Max touches on some of this stuff with a great deal of accuracy (I'm told) and equal amounts of empathy. It's quite good.

Re: AI models miss disease in Black and female patients

#117

Post by skyyler »

nonethewiser wrote: post #85
matthewmacleod wrote: post #77

Earlier quoted context omitted.

The GP literally said “giving any medical prominence to gender identity will result in people receiving wrong and potentially harmful treatment” which is categorically false for the reasons the comment you replied to outlined. Sex assigned at birth is in many situations important medical information; the vast majority of trans people are very conscious of their health in this sense and happy to share that with their…

>Sex assigned at birth is in many situations important medical information Which is not gender identity. As a result of being trans there may be things like hormone levels that are different than what you'd expect based on biological sex, which is why I say hormone levels are important, but how you identify is in fact irrelevant.

>why I say hormone levels are important, but how you identify is in fact irrelevant

I don't understand what your issue with it is, it's just another point of data.

I don't want to be treated like a cis woman in a medical context, but I sure do want to be treated like a trans woman.

Re: AI models miss disease in Black and female patients

#118

Post by kcorbitt »

ideamotor wrote: post #58
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

I really can’t help but think of the simulation hypothesis. What are the chances this copy-cat technology was developed when I was alive, given that it keeps going.

We may be in a simulation, but your odds of being alive to see this (conditioned on being born as a human at some point) aren't that low. Around 7% of all humans ever born are alive today!

Re: AI models miss disease in Black and female patients

#119

Post by cratermoon »

mhuffman wrote: post #41
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

> Can it just tell which x-rays belong to Black or female patients and then use some latent racism or misogyny to change the diagnosis?

The opposite. The dataset is for the standard model "white male", and the diagnoses generated pattern-matched on that. Because there's no gender or racial information, the model produced the statistically most likely result for white male, a result less likely to be correct for a patient that doesn't fit the standard model.

Re: AI models miss disease in Black and female patients

#120

Post by dekhn »

KittenInABox wrote: post #113
dekhn wrote: post #108

Earlier quoted context omitted.

It doesn't seem surprising at all. Genetic history correlates with race, and genetic history correlates with body-level phenotypes; race also correlates with socioeconomic status which correlates with body-level phenotypes. They are of course fairly complex correlations with many confounding factors and uncontrolled variables. It has been controversial to discuss this and a lot of discussions about this end up in fla…

What is the body-level phenotype of a ribcage by race? I think what baffles me is that black people as a group are more genetically diverse than every other race put together so I have no idea how you would identify race by ribcage x-rays exclusively.

I use the term genetic history, rather than race, as race is only weakly correlated with body level phenotypes.

If your question is truly in good faith (rather than a "I want to get in argument "), then my answer is: it's complicated. Machine learning models that work on images learn extremely complicated correlations between pixels and labels. If on average, people with a specific genetic history had slightly larger ribcages (due to their genetics, or even socioeconomic status that correlated with genetic history), that would exhibit in a number of ways in the pixels of a radiograph- larger bones spread across more pixels, density of bones slightly higher or lower, organ size differences, etc.

It is true that Africa has more genetic diversity than anywhere else; the current explanation is that after humans arose in africa, they spread and evolved extensively, but only a small number of genetically limited groups left africa and reproduced/evolved elsewhere in the world.

Post reply on HN