Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

151–160 of 256 posts

Re: AI models miss disease in Black and female patients

#151

Post by multjoy »

MountainArras wrote: post #90
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

The key takeaway from the article is that the race etc. of the subjects wasn't disclosed to the AI, yet it was able to predict it to 80% while the human experts managed 50% suggesting that there was something else encoded in the imagery that the AI was picking up on.

Re: AI models miss disease in Black and female patients

#152

Post by echoangle »

KittenInABox wrote: post #135
dekhn wrote: post #120

Earlier quoted context omitted.

I use the term genetic history, rather than race, as race is only weakly correlated with body level phenotypes. If your question is truly in good faith (rather than a "I want to get in argument "), then my answer is: it's complicated. Machine learning models that work on images learn extremely complicated correlations between pixels and labels. If on average, people with a specific genetic history had slightly larger…

I am genuinely asking because it makes no sense to me that a genetically diverse group are distinctly identifiable by their ribcage bones in an x-ray. If it's something more specific like AI sucks at statistically larger ribcages, statistically noticeable bone densities, or similar, okay. But something like so-small-humans-cannot-tell-but-is-simultaneously-widely-applicable-to-a-large-genetic-population is utterly ba…

> it makes no sense to me that a genetically diverse group are distinctly identifiable by their ribcage bones in an x-ray

I don't see how diversity would prevent identification. Butterflies are very diverse, but I still recognize one and don't think it's a bird. As long as the diversity is constrained to specific features, it can still be discriminated (and even if it's not, it technically still could be by just excluding everything else).

Re: AI models miss disease in Black and female patients

#153

Post by XorNot »

nonethewiser wrote: post #19

Race and gender should be inputs then. The female part is actually a bit more surprising. Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. But female? Thats ~52% globally.

Why not socioeconomic status or place of residence? Knowing mean yearly income will absolutely help an AI figure out statistically likely health outcomes.

Re: AI models miss disease in Black and female patients

#154

Post by mjevans »

multjoy wrote: post #151
MountainArras wrote: post #90

Earlier quoted context omitted.

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

The key takeaway from the article is that the race etc. of the subjects wasn't disclosed to the AI, yet it was able to predict it to 80% while the human experts managed 50% suggesting that there was something else encoded in the imagery that the AI was picking up on.

The AI might just have a better subjective / analytical weight detection criteria. Humans are likely more willing to see what they (or not see what they don't) expect to see.

Re: AI models miss disease in Black and female patients

#155

Post by JumpCrisscross »

bko wrote: post #67
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

Suppose you have a system that saves 90% of lives on group A but only 80% of lives in group B. This is due to the fact that you have considerably more training data on group A. You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A. So the obvious thing to do is to have the technology intentionally kill ~1 out of every 10 patients from group A so the efficacy…

> You cannot release this life saving technology because it has a 'disparate impact' on group B relative to group A

I think the point is you need to let group B know this tech works less well on them.

Re: AI models miss disease in Black and female patients

#156

Post by jfengel »

encipriano wrote: post #139
kcorbitt wrote: post #118

Earlier quoted context omitted.

We may be in a simulation, but your odds of being alive to see this (conditioned on being born as a human at some point) aren't that low. Around 7% of all humans ever born are alive today!

I dont believe that percentage. Especially considering how spread the homo branch already was more than 100 000 years ago. And from which point do you start counting? Homo erectus?

It kinda doesn't matter where you start counting. Exponential curves put almost everything at the end. Adding to the left side doesn't change it much.

You could go back to Lucy and add only a few million. Compared to the billions at this specific instant, it just doesn't make a difference.

Re: AI models miss disease in Black and female patients

#157

Post by xboxnolifes »

nottorp wrote: post #95

> as well in those 40 years or younger Are we sure it's only about racial bias then? Looks to me like the training data set is too small overall. They had too few black people, too few women, but also too few younger people.

It's the same old story that's been occurring for years/decades. Bad data in, bad data out.

Re: AI models miss disease in Black and female patients

#158

Post by ToValueFunfetti »

kcorbitt wrote: post #118
ideamotor wrote: post #58

Earlier quoted context omitted.

I really can’t help but think of the simulation hypothesis. What are the chances this copy-cat technology was developed when I was alive, given that it keeps going.

We may be in a simulation, but your odds of being alive to see this (conditioned on being born as a human at some point) aren't that low. Around 7% of all humans ever born are alive today!

In order to address the chances of a human being alive to witness the creation of this tech, you'd have to factor in the humans who have yet to be born. If you're a doomer, 7% is probably still fine. If we just maintain the current population for another century, it'll be much lower.

Re: AI models miss disease in Black and female patients

#159

Post by dekhn »

KittenInABox wrote: post #135
dekhn wrote: post #120

Earlier quoted context omitted.

I use the term genetic history, rather than race, as race is only weakly correlated with body level phenotypes. If your question is truly in good faith (rather than a "I want to get in argument "), then my answer is: it's complicated. Machine learning models that work on images learn extremely complicated correlations between pixels and labels. If on average, people with a specific genetic history had slightly larger…

I am genuinely asking because it makes no sense to me that a genetically diverse group are distinctly identifiable by their ribcage bones in an x-ray. If it's something more specific like AI sucks at statistically larger ribcages, statistically noticeable bone densities, or similar, okay. But something like so-small-humans-cannot-tell-but-is-simultaneously-widely-applicable-to-a-large-genetic-population is utterly ba…

I dunno. My perspective is that I've worked in ML for 30+ years now and over time, unsupervised clustering and direct featurization (IE, treating the image pixel as the features, rather than extracting features) have shown great utility in uncovering subtle correlations that humans don't notice. Sometimes, with careful analysis, you can sort of explain these ("it turns out the unlabelled images had the name of the hospital embedded in them, and hospital 1 had more cancer patients than hospital 2 patients because it was a regional cancer center, so the predictor learned to predict cancer more often for images that came from hospital 1") while other cases, no human, even a genius, could possibly understand the combination of variables that contributed to an output (pretty much anything in cellular biology, where billions of instances of millions of different factors act along with feedback loops and other regulation to produce systems that are robust to perturbations).

I concluded long ago I wasn't smart enough to understand some things, but by using ML, simulations, and statistics, I could augment my native intelligence and make sense of complex systems in biology. With mixed results- I don't think we're anywhere close to solving the generalized genotype to phenotype problem.

Re: AI models miss disease in Black and female patients

#160

Post by satvikpendem »

ericmcer wrote: post #115
pelorat wrote: post #107

Earlier quoted context omitted.

I think the model needs to be thought about human anatomy, not just fed a bunch of scans. It needs to understand what ribs and organs are.

I don't think LLMs can achieve "understanding" in that sense.

Computer vision models are not large language models; LLM does not mean generative AI or even AI in general, it stands for a specific initialism.
Post reply on HN