[dead]
AI models miss disease in Black and female patients
11–20 of 256 posts
Re: AI models miss disease in Black and female patients
#12This seems like a problem that should be worked on It also seems like we shouldn't let it prevent all AI deployment in the interim. It is better that we take the disease detection rate for part of the population up a few percent than we do not. Plus it's not like doctors or radiologists always diagnose at perfectly equal accuracy across all populations. Let's not let the perfect become the enemy of the good.
[flagged]
Just opt out blacks and females no?
Re: AI models miss disease in Black and female patients
#13Humans do the same. Everything from medical studies to doctor trainings treat the straight white man as the "default human" and this obviously leads to all sorts of issues. Caroline Criado-Perez has an entire chapter about this in her book about systemic bias Invisible Women, with a scary number of examples and real world consequences. It's no surprise that AI training sets reflect this also. People have been warning…
Everybody knows that gay men have more livers and fewer kidneys than straight men
Re: AI models miss disease in Black and female patients
#14I came across a fascinating Microsoft research paper on MedFuzz (https://www.microsoft.com/en-us/research/blog/medfuzz-explor...) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers.
For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaundice, bone pain, lab results) lead to “Sickle cell disease” as the correct diagnosis. However, under MedFuzz, an “attacker” LLM repeatedly modifies the question—adding information like low-income status, a sibling with alpha-thalassemia, or the use of herbal remedies—none of which should change the actual diagnosis. These additional, misleading hints can trick the “target” LLM into choosing the wrong answer. The paper highlights how real-world complexities and stereotypes can significantly reduce an LLM’s performance, even if it initially scores well on a standard benchmark.
Disclaimer: I work in Medical AI and co-founded the AI Health Institute (https://aihealthinstitute.org/).
Re: AI models miss disease in Black and female patients
#15"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.”
https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go...
Re: AI models miss disease in Black and female patients
#16[dead]
Re: AI models miss disease in Black and female patients
#17just giving globs of training sets and letting a process cook for a few months is just going to be seen as lazy in the near future more specialization of models is necessary, now that there is awareness
[dead]
Re: AI models miss disease in Black and female patients
#18Humans do the same. Everything from medical studies to doctor trainings treat the straight white man as the "default human" and this obviously leads to all sorts of issues. Caroline Criado-Perez has an entire chapter about this in her book about systemic bias Invisible Women, with a scary number of examples and real world consequences. It's no surprise that AI training sets reflect this also. People have been warning…
Everybody knows that gay men have more livers and fewer kidneys than straight men
Why the snark? The OP, the study I linked and the book I referenced which contains many well researched examples of issues caused by defaultism surely represent a strong enough body of work that they should deserve a more engaged critique.
Re: AI models miss disease in Black and female patients
#19Race and gender should be inputs then.
The female part is actually a bit more surprising. Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. But female? Thats ~52% globally.
Re: AI models miss disease in Black and female patients
#20This seems like a problem that should be worked on It also seems like we shouldn't let it prevent all AI deployment in the interim. It is better that we take the disease detection rate for part of the population up a few percent than we do not. Plus it's not like doctors or radiologists always diagnose at perfectly equal accuracy across all populations. Let's not let the perfect become the enemy of the good.
[flagged]
This is obviously a jab, but to answer the unspoken question: if it's still more effective than human doctors for black or female people, then yes it should be used. If it isn't, then don't use it for them. Simple as that. (fixing this problem should be a high priority either way, that goes without saying)