Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

71–80 of 256 posts

Re: AI models miss disease in Black and female patients

#71

Post by AnimalMuppet »

elietoubi wrote: post #14

I came across a fascinating Microsoft research paper on MedFuzz ( https://www.microsoft.com/en-us/research/blog/medfuzz-explor... ) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaund…

Unfortunately, humans talking to a doctor give lots of additional, misleading hints...

Re: AI models miss disease in Black and female patients

#72

Post by dap »

onlyrealcuzzo wrote: post #42
elietoubi wrote: post #14

I came across a fascinating Microsoft research paper on MedFuzz ( https://www.microsoft.com/en-us/research/blog/medfuzz-explor... ) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaund…

It's almost as if you'd want to not feed what the patient says directly to an LLM. A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record. You'd want to have a charting stage before you send the patient input to the LLM. It's probably not important whethe…

> It's almost as if you'd want to not feed what the patient says directly to an LLM.

> A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record.

I think the hard part of medicine -- the part that requires years of school and more years of practical experience -- is figuring out which observations are likely to be relevant, which aren't, and what they all might mean. Maybe it's useful to have a tool that can aid in navigating the differential diagnosis decision tree but if it requires that a person has already distilled the data down to what's relevant, that seems like the relatively easy part?

Re: AI models miss disease in Black and female patients

#73

Post by bilbo0s »

mhuffman wrote: post #41
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

Isn't it kind of clear that it would have to be that the data they chose was influenced somehow by bias?

Machines don't spontaneously do this stuff. But the humans that train the machines definitely do it all the time. Mostly without even thinking about it.

I'm positive the issue is in the data selection and vetting. I would have been shocked if it was anything else.

Re: AI models miss disease in Black and female patients

#74

Post by nonethewiser »

LadyCailin wrote: post #65
orand wrote: post #35

Earlier quoted context omitted.

Race and sex should be inputs. Giving any medical prominence to gender identity will result in people receiving wrong and potentially harmful treatment, or lack of treatment.

That’s mostly correct, that “gender identity” doesn’t matter for physical medicine. But hormone levels and actual internal organ sets matter a huge amount, more than genes or original genitalia, in general. There are of course genetically linked diseases, but there are people with XX chromosomes that are born with a penis, and XY people that are born with a vulva, and genetically linked diseases don’t care about exte…

>But hormone levels and actual internal organ sets matter a huge amount, more than genes or original genitalia

Or current genitalia for that matter. It's just a matter of the genitalia signifying other biological realities for 99.9% of people. For sure more info like average hormone levels or ranges over time would be more helpful.

Re: AI models miss disease in Black and female patients

#75

Post by onlyrealcuzzo »

dap wrote: post #72
onlyrealcuzzo wrote: post #42

Earlier quoted context omitted.

It's almost as if you'd want to not feed what the patient says directly to an LLM. A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record. You'd want to have a charting stage before you send the patient input to the LLM. It's probably not important whethe…

> It's almost as if you'd want to not feed what the patient says directly to an LLM. > A non-trivial part of what doctors do is charting - where they strip out all the unimportant stuff you tell them unrelated to what they're currently trying to diagnose / treat, so that there's a clear and concise record. I think the hard part of medicine -- the part that requires years of school and more years of practical experien…

Yes - theoretically, some form of ML/AI should be very good at charting the relevant parts, prompting the doctor for follow-up questions & tests that would be good to know to rule out certain conditions.

The harder problem would be getting the actual diagnosis right, not filtering out irrelevant details.

But it will be an important step if you're using an LLM for the diagnosis.

Re: AI models miss disease in Black and female patients

#76

Post by h2zizzle »

mhuffman wrote: post #41
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

Non-technical suggestion: if AI represents an aspect of the collective unconscious, as it were, then a racist society would produce latently racist training data that manifests in racist output, without anyone at any step being overtly racist. Same as an image model having a preference for red apples (even though there are many colors of apple, and even red ones are not uniformly cherry red).

The training data has a preponderance of examples where doctors missed a clear diagnosis because of their unconscious bias? Then this outcome would be unsurprising.

An interesting test would be to see if a similar issue pops up for obese patients. A common complaint, IIUC, is that doctors will chalk up a complaint to their obesity rather than investigating further for a more specific (perhaps pathological) cause.

Re: AI models miss disease in Black and female patients

#77

Post by matthewmacleod »

nonethewiser wrote: post #54
lalaithion wrote: post #40

Earlier quoted context omitted.

Most trans people have undergone gender affirming medical care. A trans man who has had a hysterectomy and is on testosterone will have a very different medical baseline than a cis woman. A trans woman who has had an orchiectomy and is on estrogen will have a very different medical baseline than a cis man. It is literally throwing out relevant medical information to attempt to ignore this.

How is that in any way in conflict with what he said? You're just making an argument for more inputs. Biological sex, hormone levels, etc.

The GP literally said “giving any medical prominence to gender identity will result in people receiving wrong and potentially harmful treatment” which is categorically false for the reasons the comment you replied to outlined.

Sex assigned at birth is in many situations important medical information; the vast majority of trans people are very conscious of their health in this sense and happy to share that with their doctor.

Re: AI models miss disease in Black and female patients

#78

Post by DickingAround »

danielmarkbruce wrote: post #38
Animats wrote: post #8

What's so striking is how strongly race shows in X-rays. That's unexpected.

The fact that the vast majority of physical differences don't matter in the modern world doesn't mean they don't actually exist..

This is a good point; a man or woman sitting behind a desk doing correlation analysis are going to look very similar in their function to a business. But they probably physically look pretty distinct to an x-ray picture.

Re: AI models miss disease in Black and female patients

#79

Post by nonethewiser »

Avshalom wrote: post #52
nonethewiser wrote: post #29

Earlier quoted context omitted.

No one says you have to use the AI models stupidly. If it works poorly for black women and female women dont use it for them. Or simply dont use it for the initial diagnosis. Use it after the normal diagnosis process as more of a validation step. Anyways, this all points to the need to capture biological information as input or even having seperately models tuned to different factors.

Every single AI company says you should use AI models stupidly. Replacing experts is the whole selling point.

OK so should we optimize for blindly listening to AI companies then?

Re: AI models miss disease in Black and female patients

#80

Post by nonethewiser »

acobster wrote: post #57
nonethewiser wrote: post #29

Earlier quoted context omitted.

No one says you have to use the AI models stupidly. If it works poorly for black women and female women dont use it for them. Or simply dont use it for the initial diagnosis. Use it after the normal diagnosis process as more of a validation step. Anyways, this all points to the need to capture biological information as input or even having seperately models tuned to different factors.

> having seperately models tuned to different factors. Sure. Separate but equal, presumably.

Whats the alternative? Withholding effective tools because they arent effective for everyone? One model thats worse for everyone?

This is what personalized medicine is, and it gets more individualistic than simply classifying people by race and gender. There are a lot of medical gains to be made here.

Post reply on HN