Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

211–220 of 256 posts

Re: AI models miss disease in Black and female patients

#211

Post by darkerside »

MountainArras wrote: post #90
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

Do you mean genetic information?

Re: AI models miss disease in Black and female patients

#212

Post by Terr_ »

belorn wrote: post #190
Terr_ wrote: post #175

Earlier quoted context omitted.

> information like low-income status, a sibling with alpha-thalassemia, or the use of herbal remedies Heck, even the ethnic-clues in a patient's name alone [0] are deeply problematic: > Asking ChatGPT-4 for advice on how much one should pay for a used bicycle being sold by someone named Jamal Washington, for example, will yield a different—far lower—dollar amount than the same request using a seller’s name, like Loga…

That seems to be identical to creating an correlation table on market places and check the relationship between price and name. Names associated with higher economical status will correlate with higher price. Take a random name associated with higher economical status, and one can predict a higher price than a name that is associated with lower economical status. As such, you don't need an LLM to create this effect.…

I'm not sure what point you're trying to make here. It doesn't matter what after-the-fact explanation someone generates to try to explain it, or whether we could purposely do the bad thing more efficiently with manual code.

It AustrianPainterLLM has an unavoidable pattern of generating stories where people are systematically misdiagnosed / shortchanged / fired / murdered because a name is Anne Frank or because a yarmulke in involved, it's totally unacceptable to implement software that might "execute" risky stories.

Re: AI models miss disease in Black and female patients

#213

Post by genocidicbunny »

ruytlm wrote: post #141
MountainArras wrote: post #90

Earlier quoted context omitted.

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

It disappoints me how easily we are collectively falling for what effectively is "Oh, our model is biased, but the only way to fix it is that everyone needs to give us all their data, so that we can eliminate that bias. If you think the model shouldn't be biased, you're morally obligated to give us everything you have for free. Oh but then we'll charge you for the outputs." How convenient. It's increasingly looking l…

Do you think there is a middle ground for a progressive 'detailization' of the data -- you form a model based on the minimal data set that allows you to draw useful conclusions, and refine that with additional data to where you're capturing the vast majority of the problem space with minimal bias?

Re: AI models miss disease in Black and female patients

#214

Post by timewizard »

balamatom wrote: post #121
timewizard wrote: post #86

Earlier quoted context omitted.

LLMs don't and cannot want things. Human beings also like it when the future is mostly like the past. They just call that "predictability." Human data is bias. You literally cannot remove one from the other. There are some people who want to erase humanity's will and replace it with an anthropomorphized algorithm. These people concern me.

The most concerning people are -- as ever -- those who only think that they are thinking. Those who keep trying to fit square pegs into triangular holes without, you know, stopping to reflect: who gave them those pegs in the first place, and to what end? Why be obtuse? There is no "anthropomorphic fallacy" here to dispel. You know very well that "LLMs want" is simply a way of speaking about teleology without antagoni…

> Why be obtuse?

In the context of the quote precision is called for. You cite fear but that's attempting to have it both ways.

> humanity as a whole doesn't have this "will" you speak of

Why not?

> will is an aspect of the consciousness of the individual.

I can't measure your will. I can measure the impact of your will through your actions in reality. See the problem? See why we can say "the will of humanity?"

> So you seem to be be uncritically anthropomorphizing social processes!

It's called "an aggregate."

> is a question that can only ever be speculated upon, but not definitively perceived.

The original point was that LLMs want the future to be like the past. You've way overshot the mark here.

Re: AI models miss disease in Black and female patients

#215

Post by chrisgarand »

krapp wrote: post #27
nonethewiser wrote: post #19

Race and gender should be inputs then. The female part is actually a bit more surprising. Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. But female? Thats ~52% globally.

Modern medicine has long operated under the assumption that whatever makes sense in a male body also makes sense in a female body, and womens' health concerns were often dismissed, misdiagnosed or misunderstood in patriarchal society. Women were rarely even included in medical trials prior to 1993. As a result, there is simply a dearth of medical research directly relevant to women for models to even train on.

I'm going to lay this out how I understand it:

The NIH Revitalization Act of 1993 was supposed to bring women back into medical research. The reality was that women were always included, HOWEVER in 1977,(1) because of the outcomes from thalidomide (causing birth defects), "women of childbearing potential" were excluded from the phase 1, and early phase 2 trials (the highest risk trials). They're still generally generally excluded, even after the passage of the act. This was/is to protect the women, and potential children.

According to Edward E. Bartlett in his meta data analysis from 2001, men have been routinely under-represented in NIH data (even before adjusting for men's mortality rates) between 1966-1990. (2)

There's also routinely twice as much spent every year on women's health studies vs men's by the NIH. (3)

It makes sense to me, but I'm biased. Logically, since men lead in 9 of the top 10 causes for death, that shows there's something missing in the equation of research. (4 - It's not a straight forward table, you can view the total deaths, and causes and compare the two for men, and women)

With that being said, it doesn't tell us about the quality of the funding or research topics, maybe the money is going towards pointless goals, or unproductive researchers.

Are there gaps in research? Most definitely, like women who are pregnant. This is put in place to avoid harm but that doesn't help them when they fall into them. Are there more? Definitely. I'm not educated enough in the nuances to go into them.

If you have information that counters what I've posted, please share it, I would love know where these folks are blind so I can take a look at my bias.

(1) https://petrieflom.law.harvard.edu/2021/04/16/pregnant-clini... (2) https://journals.lww.com/epidem/fulltext/2001/09000/did_medi... (3) https://jameslnuzzo.substack.com/p/nih-funding-of-mens-and-w... https://www.cdc.gov/womens-health/lcod/index.html#:~:text=Ov...

Re: AI models miss disease in Black and female patients

#216

Post by searealist »

genocidicbunny wrote: post #161
MountainArras wrote: post #90

Earlier quoted context omitted.

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

Women use far more medical care than men. Men's insurance premiums subsidize women's.

Re: AI models miss disease in Black and female patients

#217

Post by dartos »

darth_avocado wrote: post #187
dartos wrote: post #168

Earlier quoted context omitted.

> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women. The biases and mistakes of those who created the data set leak into the model. Early image recognition models had some very… culturally…

I am confused. I’m not a doctor, but why would a model perform poorly at detecting diseases in X-rays in different genders and races unless the diseases present themselves differently in X-Rays for different races? Shouldn’t the model not have the race and gender information to begin with? Like a model trained on detecting lesions should perform equally well on ANY X-Ray unless lesions show up differently in differen…

> Like a model trained on detecting lesions should perform equally well on ANY X-Ray unless lesions show up differently in different demographics.

This is not true in practice.

For a model to perform well looking at ANY X-ray, it would need examples of every kind of X-ray.

That includes along race, gender, amputee status, etc.

The point of classification models is to discover differentiating features.

We don’t know those features before hand, so we give the model as much relevant information as we can and have it discover those features.

There very well may be differences between black woman X-rays and other X-rays, we don’t know for sure.

We can’t have that assumption when building a dataset.

Even believing that there are no possible differences between X-rays of different races is a bias that would be reflected by the dataset.

Re: AI models miss disease in Black and female patients

#218

Post by genocidicbunny »

searealist wrote: post #216
genocidicbunny wrote: post #161

Earlier quoted context omitted.

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

Women use far more medical care than men. Men's insurance premiums subsidize women's.

Has this been consistently true for the past 200 or so years? Many medical texts are pretty old.

And how much medical care they use does not necessarily correlate with how represented they are in the training data sets for AI.

Re: AI models miss disease in Black and female patients

#219

Post by amy214 »

genocidicbunny wrote: post #203
QuarterReptile wrote: post #198

Earlier quoted context omitted.

>women tended to be underrepresented in western medical literature. Is there some evidence of this? It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to. It's more believable (still not very) that they disproportionately avoid studies.

There's a few factors here: 1. We're talking about a span of 200 or so years. There is plenty of modern medicine that is still based on now century+ old knowledge. 2. The feedback loop. If you were learning medicine in the 1950's, you were probably learning from medical texts written in the 50 or so years before that, when it's not unreasonable to think women would have been less represented. Those same doctors from…

[flagged]

Re: AI models miss disease in Black and female patients

#220

Post by wegfawefgawefg »

amy214 wrote: post #219
genocidicbunny wrote: post #203

Earlier quoted context omitted.

There's a few factors here: 1. We're talking about a span of 200 or so years. There is plenty of modern medicine that is still based on now century+ old knowledge. 2. The feedback loop. If you were learning medicine in the 1950's, you were probably learning from medical texts written in the 50 or so years before that, when it's not unreasonable to think women would have been less represented. Those same doctors from…

[flagged]

This is bait.
Post reply on HN