Live data from Hacker News

AI models miss disease in Black and female patients

science.org

231–240 of 256 posts

Re: AI models miss disease in Black and female patients

#231

Earlier quoted context omitted.

Proof that health insurance premiums for men have been consistently subsidizing women's health insurance premiums for the last 200 or so years? Perhaps the practical non-existence of health insurance until the latter half of the 20th's century? Pretty tough to subsidize something that doesn't exist. You also offered no evidence for your assertion in the first place.

The ACA bans health insurance companies from charging men and women different rates for the same coverage. Before this, Women would have higher premiums because, on average, they use their coverage more. This is very easy to look up. I can cite the ACA, but you can not cite anything that says AI training sets are biased against women.

A few questions for you to think of then -- or rather a few things I think you should consider with your statements:

1. How does ACA affect the corpus of knowledge and medical practice gathered prior to the ACA being in effect? How does it affect late 19th, and early and mid 20th century medical knowledge and practice, which occurred prior to health insurance of any kind, nevermind ACA-compliant, being widespread? This corpus of knowledge and practice continues to propagate even now. I've read a handful of recently published medical textbooks and there are definitely parts that are pretty much the same as the textbooks of the early 20th century, just with slightly updated language.

2. What are the possible confounding factors in the use of health insurance by men vs women? For example, could men just be more hesitant to see a doctor, and thus less likely to make use of health insurance? Does the average life expectancy of women result in more use of health insurance later in life than for men? Are medical procedures that are specific to women that add to the cost of their care, such as mammograms, pap smears, etc? Seeings as how in the US health insurance is a practical requirement to getting medical care, and lack of it is punished financially in various ways from taxes to just having medical care be more expensive when you truly need it, means most people will try to have _some_ kind of health insurance, even if they don't think they need it for actual health reasons. So despite a perception of not needing health insurance, men are incentivized to have health insurance they don't use?

3. Does the ACA guarantee in any way that medical professionals no longer hold any bias due their previous training, especially if such training occurred prior to the introduction of the ACA? Does the ACA similarly guarantee that women and men are not only able, but choose to pursue medical care and participate in medical studies at percentages matching the general population?

Your point about men subsidizing women with regard to health insurance premiums may be perfectly valid, I am not disputing you on that point. I am disputing that it is salient to the tradition and practice of medicine in the western world in the modern era, until very recently historically, and that these traditions and biases will affect data sets gathered from people who are directly affected by these biases and traditions to this day. We haven't eliminated them, because as I said in another comment, every generation just dilutes the old issues, it doesn't solve them. And while I could spend my evening finding studies from various countries that attest to my view on this, I have spent about as much time as I desire to on this, so I will grant you that my evidence is on the level of 'trust me bro' -- with the slight caveat that many people within just my family and close circle of friends are involved in the medical field and all largely agree to this, and they are not all based in the US (which by the way, your point is very specific to. ACA is a US thing, western medicine spans a bit more than that.) It is entirely fair for you to call out that I have offered no real peer-reviewed evidence for my statements. I intend to offer a viewpoint of someone who has had extensive peripheral experience with medical professionals and has discussed this topic with them, and to offer some avenues of thought on how and why the data sets might be biased.

Re: AI models miss disease in Black and female patients

#232
post #35

Earlier quoted context omitted.

Race and sex should be inputs. Giving any medical prominence to gender identity will result in people receiving wrong and potentially harmful treatment, or lack of treatment.

Most trans people have undergone gender affirming medical care. A trans man who has had a hysterectomy and is on testosterone will have a very different medical baseline than a cis woman. A trans woman who has had an orchiectomy and is on estrogen will have a very different medical baseline than a cis man. It is literally throwing out relevant medical information to attempt to ignore this.

[flagged]

Re: AI models miss disease in Black and female patients

#233

Earlier quoted context omitted.

One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…

>women tended to be underrepresented in western medical literature. Is there some evidence of this? It's hard for me to picture that women see receive less medical attention than man: completely inconsistent with my culture and every doctor's office I've ever been to. It's more believable (still not very) that they disproportionately avoid studies.

Here is an academic medicine perspective: https://www.aamc.org/news/why-we-know-so-little-about-women-...

To give you some TL;DR from personal-ish experience, women have historically been excluded from medical trials because:

* why include them? people are people, right? * except when they're pregnant or could be pregnant -- a trial by definition has risks, and so "of course" one would want to exclude anyone who is or could get pregnant (it's the clinical trial version of "she's just going to get married and leave the job anyway") * and cyclical fluctuations in hormones are annoying.

The first one is wrong (tho is an oversight that many had for years, assuming for instance that heart attacks and autism would present with the same symptoms in all adult humans).

The second is an un-nuanced approach to risk. Pregnant ladies also need medical treatment for things, and it's pretty annoying to be pregnant and be told that you need to decide among unstudied treatments for some non-pregnancy-related problem.

The third is just a difficult fact of life. I know researchers studying elite performance in women athletes, for instance. At an elite level, it would be useful to understand if there are different effects of training (strength, speed, endurance) at different times in the menstrual cycle. To do this, you need to measure hormone levels in the blood to establish on a scientific basis where in the cycle a study participant is. Turns out there is significant heterogeneity in how this process works. So some scientists in the field are arguing that studies should only be conducted on women who are experiencing "normal menstrual cycles" which is defined by them as three continuous months of a cycle between 28-35 days. So to establish that then you've got to get these ladies in for three months before the study can even start, getting these hormone levels measured to establish that the cycle is "normal", before you can even start your intervention. (Ain't no one got $$ for that...) And that's before we bring in the fact that many women performing on an elite level in sport don't have a normal menstrual cycle. But from the sports side, they'd still like to know what training is most effective.... so that's a very current debate in the field. And I haven't even started on hormonal birth control! Birth control provides a base level of hormone circulating in the blood, but if it's from a pill it's varying on a daily basis, while if it's a patch or ring it's on a monthly basis (or longer). There's some question of whether that hormonal load from the birth control is then suppressing natural production of some hormones. And why does this matter? Because estrogen for instance has significant effects on cardiovascular health, being cardioprotective from puberty up to menopause. (Yeah, I didn't even get started on perimenopause or menopause.)

Fine, fine, it's just data analysis & logistics. If you get the ladies (only between 21-35) into the lab for blood samples frequently enough and measure at the same time of day every time to avoid daily effects and find a large enough group that you can dump all the ladies who don't fit some definition of normal & anyone who gets pregnant but still get the power for your study, it's all fine, right? You've just expanded medical research to incorporate, like, 10% more of the population....!

Re: AI models miss disease in Black and female patients

#234
The article addresses much of what is incorrectly speculated on in the comments here.

“Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses” (including demographics).

Re: AI models miss disease in Black and female patients

#235
post #168

Earlier quoted context omitted.

> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women. The biases and mistakes of those who created the data set leak into the model. Early image recognition models had some very… culturally…

Xays by definition don't look at skin color. Do chest x-rays of black women reveal that there's something different about their chests than white or asian women? That doesn't pass my non doctor sniff test, but someone can correct me (no sarcasm intended).

Breast density affects the imaging you get from x-rays. It is well-known that denser breast tissue results in x-rays that are "whiter" (I'm talking about the image of the tissue, in white, on a black background, as x-rays are commonly read by radiologists). Denser breasts are associated with less effective screening for breast cancer via mammogram. A mammogram is a low-dose x-ray.

When using a chest x-ray to look for pulmonary edema, for instance, I would be unsurprised if breast tissue (of any quantity) and in particular denser breast tissue would make the diagnosis of pulmonary edema more difficult from the image alone.

Also, you seem to have conflated a few things in your second sentence. Deep in the article, they did have radiologists try to guess demographic attributes by looking at the x-ray images. They were pretty good at guessing female/male (unsurprising) and were not really able to guess age or race. So I'm super interested in how the AI model was able to be better at that than the human radiologists.

Re: AI models miss disease in Black and female patients

#236

Earlier quoted context omitted.

The ACA bans health insurance companies from charging men and women different rates for the same coverage. Before this, Women would have higher premiums because, on average, they use their coverage more. This is very easy to look up. I can cite the ACA, but you can not cite anything that says AI training sets are biased against women.

A few questions for you to think of then -- or rather a few things I think you should consider with your statements: 1. How does ACA affect the corpus of knowledge and medical practice gathered prior to the ACA being in effect? How does it affect late 19th, and early and mid 20th century medical knowledge and practice, which occurred prior to health insurance of any kind, nevermind ACA-compliant, being widespread? Th…

Women using more health care didn't start with the ACA. The ACA just banned the practice of charging women more because they use more health care.

Ask a doctor what gender goes to them more for gender neutral health care like "flu-like symptoms".

Now you provide evidence that AI models discriminate against Women instead of DDoSing me with "how can you know its not true" written in 10 ways.

Funny how you never read a headline about how Latinos or Asians are discriminated against in medical science. That's a pretty clear give away that this is politically motivated.

Are you going to hold the same standard to them? Were Asians and Latinos represented in 200 year old medical texts?

Re: AI models miss disease in Black and female patients

#237
post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

[deleted]

Re: AI models miss disease in Black and female patients

#238
> Compared with the patients’ doctors, the AI model more often failed to detect the presence of disease in Black patients or women, as well in those 40 years or younger.

Garbage article. Garbage study. And garbage AI model that doesn't account for most of its audience.

Re: AI models miss disease in Black and female patients

#239

Earlier quoted context omitted.

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

> Can it just tell which x-rays belong to Black or female patients and then use some latent racism or misogyny to change the diagnosis? The opposite. The dataset is for the standard model "white male", and the diagnoses generated pattern-matched on that. Because there's no gender or racial information, the model produced the statistically most likely result for white male, a result less likely to be correct for a pat…

Then why is the headline not "AI models miss disease in Asian patients" or even "AI models miss disease in Latino patients"?

It just so happens to align with what maximizes political capital in today's world.

Re: AI models miss disease in Black and female patients

#240
To be honest, I'm not sure if I understand what is actually claimed here (it seems that they trained the model on their own, and claim that the problem is in the dataset?) but isn't the more sensible explanation that human doctors were overdiagnosing them?
Post reply on HN