Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

161–170 of 256 posts

Re: AI models miss disease in Black and female patients

#161

Post by genocidicbunny »

MountainArras wrote: post #90
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations of various ailments that may have variance due to gender or ethnicity. Basically, if most data is gathered from those who have the most access to medicine, there will be an inherent bias towards how various ailments present in those populations. So your base data set might be skewed from the very beginning.

(This is mostly just to offer some food for thought, I haven't read the article in full so I don't want to comment on it specifically.)

Re: AI models miss disease in Black and female patients

#162

Post by connicpu »

orand wrote: post #35
nonethewiser wrote: post #19

Race and gender should be inputs then. The female part is actually a bit more surprising. Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. But female? Thats ~52% globally.

Race and sex should be inputs. Giving any medical prominence to gender identity will result in people receiving wrong and potentially harmful treatment, or lack of treatment.

Actually both are important inputs, especially when someone has been taking hormones for a very long time. The human body changes greatly. Growing breast tissue increases the likelyhood of breast cancer, for example, compared to if you had never taken it (but about the same as if estradiol had been present during your initial puberty).

Re: AI models miss disease in Black and female patients

#163

Post by connicpu »

jl6 wrote: post #126
matthewmacleod wrote: post #93

Earlier quoted context omitted.

Well, this is clearly wrong – it's obvious, for example, that gender identity could have a significant impact on mental health. Regardless of that, you seem to agree that: - Sex assigned at birth is important medical information - Information about gender affirming treatments is important medical information So I don't think there's much to worry about there.

The problem is that over the past few decades there has been substantial conflation of sex and gender, with many information systems replacing the former with the latter, rather than augmenting data collection with the latter.

I think it's pretty clear to see how discrimination is the cause of that. Why would you volunteer information that from your point of view is more likely to cause a negative interaction than not?

Re: AI models miss disease in Black and female patients

#164

Post by root_axis »

lalaithion wrote: post #40
orand wrote: post #35

Earlier quoted context omitted.

Race and sex should be inputs. Giving any medical prominence to gender identity will result in people receiving wrong and potentially harmful treatment, or lack of treatment.

Most trans people have undergone gender affirming medical care. A trans man who has had a hysterectomy and is on testosterone will have a very different medical baseline than a cis woman. A trans woman who has had an orchiectomy and is on estrogen will have a very different medical baseline than a cis man. It is literally throwing out relevant medical information to attempt to ignore this.

Seems like adding in gender only makes things less clear. The relevant information is sex and a medical history of specific surgeries and medications - the type of thing your doctor should already be aware of. Adding in gender only creates ambiguity because there's no way to measure gender from a biological perspective.

Re: AI models miss disease in Black and female patients

#165

Post by bflesch »

dekhn wrote: post #159
KittenInABox wrote: post #135

Earlier quoted context omitted.

I am genuinely asking because it makes no sense to me that a genetically diverse group are distinctly identifiable by their ribcage bones in an x-ray. If it's something more specific like AI sucks at statistically larger ribcages, statistically noticeable bone densities, or similar, okay. But something like so-small-humans-cannot-tell-but-is-simultaneously-widely-applicable-to-a-large-genetic-population is utterly ba…

I dunno. My perspective is that I've worked in ML for 30+ years now and over time, unsupervised clustering and direct featurization (IE, treating the image pixel as the features, rather than extracting features) have shown great utility in uncovering subtle correlations that humans don't notice. Sometimes, with careful analysis, you can sort of explain these ("it turns out the unlabelled images had the name of the ho…

Sounds like "geoguesser" players who learn to recognize google street view pictures from a specific country by looking at the color of the google street view car or a specific piece of dirt on the camera lens.

Re: AI models miss disease in Black and female patients

#166

Post by mg794613 »

Thats bad! Let's change that! Let's be better than our predecessors! Right?

So, how do they suggest to tackle the problem?

1. Improve the science 2. Update the data

or

3. Somehow focus on it being racist and then walking away like the hero of the day without actually solving the problem.

Re: AI models miss disease in Black and female patients

#167

Post by MonkeyClub »

jhanschoo wrote: post #147
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

I like how the author used neo-Greek words to sneak in graphic imagery that would normally be taboo in this register of writing

I dislike how they misspelled it though.

Re: AI models miss disease in Black and female patients

#168

Post by dartos »

MountainArras wrote: post #90
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here.

For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women.

The biases and mistakes of those who created the data set leak into the model.

Early image recognition models had some very… culturally insensitive classes baked in.

Re: AI models miss disease in Black and female patients

#169

Post by kulahan »

echoangle wrote: post #150
elietoubi wrote: post #14

I came across a fascinating Microsoft research paper on MedFuzz ( https://www.microsoft.com/en-us/research/blog/medfuzz-explor... ) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaund…

> a sibling with alpha-thalassemia I have no clue what that is or why it shouldn't change the diagnosis, but it seems to be a genetic thing. Is the problem that this has nothing to do with the described symptoms? Because surely, a sibling having a genetic disease would be relevant if the disease could be a cause of the symptoms?

In medicine, if it walk like a horse and talks like a horse, it’s a horse. You don’t start looking into the health of relatives when your patient tells the full story on their own.

Sickle cell anemia is common among African Americans (if you don’t have the full-blown version, the genes can assist with resisting one of the common mosquito-borne diseases found in Africa, which is why it developed in the first place I believe).

So, we have a patient in the primary risk group presenting with symptoms that match well with SCA. You treat that now, unless you have a specific reason not to.

Sometimes you have a list of 10-ish diseases in order of descending likelihood, and the only way to rule out which one it isn’t, is by seeing no results from the treatment.

Edit: and it’s probably worth mentioning no patient ever gives ONLY relevant info. Every human barrages you with all the things hurting that may or may not be related. A doctor’s specific job in that situation is to filter out useless info.

Re: AI models miss disease in Black and female patients

#170

Post by acobster »

nonethewiser wrote: post #80
acobster wrote: post #57

Earlier quoted context omitted.

> having seperately models tuned to different factors. Sure. Separate but equal, presumably.

Whats the alternative? Withholding effective tools because they arent effective for everyone? One model thats worse for everyone? This is what personalized medicine is, and it gets more individualistic than simply classifying people by race and gender. There are a lot of medical gains to be made here.

I'm not arguing against using the models per se. It's just that this is a social problem, to which there's no good technical solution. The hard road of social change is the only real alternative.
Post reply on HN