Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

91–100 of 256 posts

Re: AI models miss disease in Black and female patients

#91

Post by potsandpans »

bko wrote: post #67
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

Suppose you have a system that saves 90% of lives on group A but only 80% of lives in group B. This is due to the fact that you have considerably more training data on group A. You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. So the obvious thing to do is to have the technology intentionally kill ~1 out of every 10 patients from group A so the efficacy…

Imagine if you had a strawman so full of straw, it was the most strawfilled man that ever existed.

Re: AI models miss disease in Black and female patients

#93

Post by matthewmacleod »

nonethewiser wrote: post #85
matthewmacleod wrote: post #77

Earlier quoted context omitted.

The GP literally said “giving any medical prominence to gender identity will result in people receiving wrong and potentially harmful treatment” which is categorically false for the reasons the comment you replied to outlined. Sex assigned at birth is in many situations important medical information; the vast majority of trans people are very conscious of their health in this sense and happy to share that with their…

>Sex assigned at birth is in many situations important medical information Which is not gender identity. As a result of being trans there may be things like hormone levels that are different than what you'd expect based on biological sex, which is why I say hormone levels are important, but how you identify is in fact irrelevant.

Well, this is clearly wrong – it's obvious, for example, that gender identity could have a significant impact on mental health.

Regardless of that, you seem to agree that:

- Sex assigned at birth is important medical information

- Information about gender affirming treatments is important medical information

So I don't think there's much to worry about there.

Re: AI models miss disease in Black and female patients

#94

Post by nonethewiser »

Avshalom wrote: post #84
nonethewiser wrote: post #79

Earlier quoted context omitted.

OK so should we optimize for blindly listening to AI companies then?

We should assume people will use tools in the manner that they have been sold those tools yes.

But these tools include research like this. This research is sold as proof that AI models have problems with bias. So by your reasoning I'd expect doctors to be wary of AI models.

Re: AI models miss disease in Black and female patients

#96

Post by bko »

janice1999 wrote: post #81
bko wrote: post #67

Earlier quoted context omitted.

Suppose you have a system that saves 90% of lives on group A but only 80% of lives in group B. This is due to the fact that you have considerably more training data on group A. You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. So the obvious thing to do is to have the technology intentionally kill ~1 out of every 10 patients from group A so the efficacy…

> You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. Who is preventing you in this imagined scenario? There are drugs that are more effective on certain groups of people than others. BiDil, for example, is an FDA approved drug marketed to a single racial-ethnic group, African Americans, in the treatment of congestive heart failure. As long as the risks a…

What does this have to do with FDA or drugs? Re-read the comment I was replying to. It's complaining that a technology could serve one group of people better than another, and I would argue that this should not be our goal.

A technology should be judged by "does it provide value to any group or harm any other group". But endlessly dividing people into groups and saying how everything is unfair because it benefits group A over group B due to the nature of the problem, just results in endless hand-wringing and conservatism and delays useful technology from being released due to the fear of mean headlines like this.

Re: AI models miss disease in Black and female patients

#98

Post by dekhn »

nradov wrote: post #62
onlyrealcuzzo wrote: post #42

Earlier quoted context omitted.

It's almost as if you'd want to not feed what the patient says directly to an LLM. A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record. You'd want to have a charting stage before you send the patient input to the LLM. It's probably not important whethe…

I generally agree, however socioeconomic and environmental factors are highly correlated with certain medical conditions (social determinants of health). In some cases even causative. For example, patients who live near an oil refinery are more likely to have certain cancers or lung diseases. https://doi.org/10.1093/jncics/pkaa088

Studies like that, no matter how careful, cannot say anything about causation.

Re: AI models miss disease in Black and female patients

#99

Post by Avshalom »

nonethewiser wrote: post #94
Avshalom wrote: post #84

Earlier quoted context omitted.

We should assume people will use tools in the manner that they have been sold those tools yes.

But these tools include research like this. This research is sold as proof that AI models have problems with bias. So by your reasoning I'd expect doctors to be wary of AI models.

doctors aren't being sold this. Private equity firms that buy hospitals are.

Re: AI models miss disease in Black and female patients

#100

Post by FanaHOVA »

mhuffman wrote: post #41
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

The non-tinfoil hat approach is to simply Google "Boston demographics", and think of how training data distribution impacts model performance.

> The data set used to train CheXzero included more men, more people between 40 and 80 years old, and more white patients, which Yang says underscores the need for larger, more diverse data sets.

I'm not a doctor so I cannot tell you how xrays differ across genders / ethnicities, but these models aren't magic (especially computer vision ones, which are usually much smaller). If there are meaningful differences and they don't see those specific cases in training data, they will always fail to recognize them at inference.

Post reply on HN