Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

81–90 of 256 posts

Re: AI models miss disease in Black and female patients

#81

Post by janice1999 »

bko wrote: post #67
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

Suppose you have a system that saves 90% of lives on group A but only 80% of lives in group B. This is due to the fact that you have considerably more training data on group A. You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. So the obvious thing to do is to have the technology intentionally kill ~1 out of every 10 patients from group A so the efficacy…

> You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A.

Who is preventing you in this imagined scenario?

There are drugs that are more effective on certain groups of people than others. BiDil, for example, is an FDA approved drug marketed to a single racial-ethnic group, African Americans, in the treatment of congestive heart failure. As long as the risks are understood there can be accommodations made ("this AI tool is for males only" etc). However such limitations and restrictions are rarely mentioned or understood by AI hype people.

Re: AI models miss disease in Black and female patients

#82

Post by bilbo0s »

bko wrote: post #67
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

Suppose you have a system that saves 90% of lives on group A but only 80% of lives in group B. This is due to the fact that you have considerably more training data on group A. You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. So the obvious thing to do is to have the technology intentionally kill ~1 out of every 10 patients from group A so the efficacy…

No. That's not how it works.

It's contraindication. So you're in a race to the bottom in a busy hospital or clinic. Where people throw group A in a line to look at what the AI says, and doctors and nurses actually look at people in group B. Because you're trying to move patients through the enterprise.

The AI is never even given a chance to fail group B. But now you've got another problem with the optics.

Re: AI models miss disease in Black and female patients

#83

Post by chadd »

A surprisingly high number of medical studies will not include women because the study doesn't want to account for "outliers" like pregnancy and menstrual cycles[0]. This is bound to have effects on LLM answers for women.

[0] https://www.northwell.edu/katz-institute-for-womens-health/a...

Re: AI models miss disease in Black and female patients

#84

Post by Avshalom »

nonethewiser wrote: post #79
Avshalom wrote: post #52

Earlier quoted context omitted.

Every single AI company says you should use AI models stupidly. Replacing experts is the whole selling point.

OK so should we optimize for blindly listening to AI companies then?

We should assume people will use tools in the manner that they have been sold those tools yes.

Re: AI models miss disease in Black and female patients

#85

Post by nonethewiser »

matthewmacleod wrote: post #77
nonethewiser wrote: post #54

Earlier quoted context omitted.

How is that in any way in conflict with what he said? You're just making an argument for more inputs. Biological sex, hormone levels, etc.

The GP literally said “giving any medical prominence to gender identity will result in people receiving wrong and potentially harmful treatment” which is categorically false for the reasons the comment you replied to outlined. Sex assigned at birth is in many situations important medical information; the vast majority of trans people are very conscious of their health in this sense and happy to share that with their…

>Sex assigned at birth is in many situations important medical information

Which is not gender identity. As a result of being trans there may be things like hormone levels that are different than what you'd expect based on biological sex, which is why I say hormone levels are important, but how you identify is in fact irrelevant.

Re: AI models miss disease in Black and female patients

#86

Post by timewizard »

orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

LLMs don't and cannot want things. Human beings also like it when the future is mostly like the past. They just call that "predictability."

Human data is bias. You literally cannot remove one from the other.

There are some people who want to erase humanity's will and replace it with an anthropomorphized algorithm. These people concern me.

Re: AI models miss disease in Black and female patients

#87

Post by NotYourLawyer »

nonethewiser wrote: post #63
NotYourLawyer wrote: post #56

[flagged]

Interesting catch. TIL I learned about this split. >At the Columbia Journalism Review, we capitalize Black, and not white, when referring to groups in racial, ethnic, or cultural terms. For many people, Black reflects a shared sense of identity and community. White carries a different set of meanings; capitalizing the word in this context risks following the lead of white supremacists. https://www.cjr.org/analysis/ca…

[flagged]

Re: AI models miss disease in Black and female patients

#88

Post by thereisnospork »

onlyrealcuzzo wrote: post #70
nradov wrote: post #62

Earlier quoted context omitted.

I generally agree, however socioeconomic and environmental factors are highly correlated with certain medical conditions (social determinants of health). In some cases even causative. For example, patients who live near an oil refinery are more likely to have certain cancers or lung diseases. https://doi.org/10.1093/jncics/pkaa088

So that's the important part, not that they're low income.

Sure, but correlation is correlation. Ergo 'low income', as well as affections or causes of being 'low income' are valid diagnostic indicators.

Re: AI models miss disease in Black and female patients

#89

Post by sdsd »

schnable wrote: post #61
sergiotapia wrote: post #51

Earlier quoted context omitted.

It's odd how we can segment between different species in animals, but in humans it's taboo to talk about this. Threw the baby out with the baby water. I hope we can fix this soon so everybody can benefit from AI. The fact that I'm a male latino should be an input for an AI trained on male latinos! I want great care! I don't want pretend kumbaya that we are all humans in the end. That's not true. We are distinct! We a…

That's because humans are all the same species.

In terms ofLinnaean taxonomy, and Chihuahuas and wolves are also the same species, in that they can reproduce fertile offspring. We instead differentiate them using the less objective subspecies classification. So it appears that with canines we're comfortable delineating subspecies, why not with humans?

I don't think we should, but your particular argument seems open to this critique.

Re: AI models miss disease in Black and female patients

#90

Post by MountainArras »

orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender data too in order to get better results when predicting the targeted diseases within these groups.

I think it's interesting to think about instead attaching generic information instead of group data, which would be blind to human bias and the messiness of our rough categorizations of subgroups.

Post reply on HN