Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

101–110 of 256 posts

Re: AI models miss disease in Black and female patients

#101

Post by nradov »

nonethewiser wrote: post #80
acobster wrote: post #57

Earlier quoted context omitted.

> having seperately models tuned to different factors. Sure. Separate but equal, presumably.

Whats the alternative? Withholding effective tools because they arent effective for everyone? One model thats worse for everyone? This is what personalized medicine is, and it gets more individualistic than simply classifying people by race and gender. There are a lot of medical gains to be made here.

Citation needed. Personalized medicine seems like a great idea in principle, but so far attempts to put it into practice have been underwhelming in terms of improved patient outcomes. You seem to be assuming that these tools actually are effective, but generally that remains unproven.

Re: AI models miss disease in Black and female patients

#103

Post by daemonologist »

cheschire wrote: post #43
elietoubi wrote: post #14

I came across a fascinating Microsoft research paper on MedFuzz ( https://www.microsoft.com/en-us/research/blog/medfuzz-explor... ) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaund…

Can't the same be said for humans though? Not to be too reductive, but aren't most general practitioners just pattern recognition machines?

I'm sure humans can make similar errors, but we're definitely less suggestible than current language models. For example, if you tell a chat-tuned LLM it's incorrect, it will almost always respond with something like "I'm sorry, you're right..." A human would be much more likely to push back if they're confident.

Re: AI models miss disease in Black and female patients

#104

Post by bko »

potsandpans wrote: post #91
bko wrote: post #67

Earlier quoted context omitted.

Suppose you have a system that saves 90% of lives on group A but only 80% of lives in group B. This is due to the fact that you have considerably more training data on group A. You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. So the obvious thing to do is to have the technology intentionally kill ~1 out of every 10 patients from group A so the efficacy…

Imagine if you had a strawman so full of straw, it was the most strawfilled man that ever existed.

From the article:

> “What is clear is that it’s going to be really difficult to mitigate these biases,” says Judy Gichoya, an interventional radiologist and informatician at Emory University who was not involved in the study. Instead, she advocates for smaller, but more diverse data sets that test these AI models to identify their flaws and correct them on a small scale first. Even so, “Humans have to be in the loop,” she says. “AI can’t be left on its own.”

What do you think smaller data sets would do to a model? It'll get rid of disparity sure

Re: AI models miss disease in Black and female patients

#106

Post by apical_dendrite »

protonbob wrote: post #49
mhuffman wrote: post #41

Earlier quoted context omitted.

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

I'm going to wager an uneducated guess. Black people are less likely to go to the doctor for both economic and historical reasons so images from them are going to be underrepresented. So in some way I guess you could say that yes, latent racism caused people to go to the doctor less which made them appear less in the data.

Where the data comes from also matters. Data is collected based on what's available to the researcher. Data from a particular city or time period may have a very different distribution than the general population.

Re: AI models miss disease in Black and female patients

#107

Post by pelorat »

MountainArras wrote: post #90
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

I think the model needs to be thought about human anatomy, not just fed a bunch of scans. It needs to understand what ribs and organs are.

Re: AI models miss disease in Black and female patients

#108

Post by dekhn »

Animats wrote: post #8

What's so striking is how strongly race shows in X-rays. That's unexpected.

It doesn't seem surprising at all. Genetic history correlates with race, and genetic history correlates with body-level phenotypes; race also correlates with socioeconomic status which correlates with body-level phenotypes. They are of course fairly complex correlations with many confounding factors and uncontrolled variables.

It has been controversial to discuss this and a lot of discussions about this end up in flamewars, but it doesn't seem surprising, at least to me, from my understanding of the relationship between genetic history and body-level phenotypes.

Re: AI models miss disease in Black and female patients

#110

Post by zeagle »

Cool topic! This isn't surprising given the AI models would be trained such that existing medical practices, biases, and failures would propagate through them as others have said here.

There is a published, recognized bias against women and blacks (borrowing the literature term) specifically in medicine when it comes to pain assessment and treatment. Racism is a part of it but too simplistic. Most of us don't go to work trying to be horrible people. I was in a fly in community earlier this week for work where 80% of housing is subsidized social housing... so spit balling a bit... things like assumptions about rate of metabolizing medications being equal, assess to medication, culture and stoicism, dismissing concerts, and the broad effects of poverty/trauma/inter-generational trauma all must play a role in this.

For interest:

https://jamanetwork.com/journals/jamanetworkopen/fullarticle...

Overall, the authors found comparable ratings in Black and White participants’ perceptions of the patient-physician relationship across all three measures (...) Alternatively, the authors found significant racial differences in the pain-related outcomes, including higher pain intensity and greater back-related disability among Black participants compared with White participants (intensity mean: 7.1 vs 5.8; P https://www.aamc.org/news/how-we-fail-black-patients-pain

(top line summary) Half of white medical trainees believe such myths as black people have thicker skin or less sensitive nerve endings than white people. An expert looks at how false notions and hidden biases fuel inadequate treatment of minorities’ pain.

And https://www.washingtonpost.com/wellness/interactive/2022/wom...

Post reply on HN