Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

141–150 of 256 posts

Re: AI models miss disease in Black and female patients

#141

Post by ruytlm »

MountainArras wrote: post #90
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…

It disappoints me how easily we are collectively falling for what effectively is "Oh, our model is biased, but the only way to fix it is that everyone needs to give us all their data, so that we can eliminate that bias. If you think the model shouldn't be biased, you're morally obligated to give us everything you have for free. Oh but then we'll charge you for the outputs."

How convenient.

It's increasingly looking like the AI business model is "rent extracting middleman", just like the Elseviers et al of the academic publishing world - wedging themselves into a position where they get to take everything for free, but charge others at every opportunity.

Re: AI models miss disease in Black and female patients

#142

Post by encipriano »

protonbob wrote: post #49
mhuffman wrote: post #41

Earlier quoted context omitted.

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

I'm going to wager an uneducated guess. Black people are less likely to go to the doctor for both economic and historical reasons so images from them are going to be underrepresented. So in some way I guess you could say that yes, latent racism caused people to go to the doctor less which made them appear less in the data.

Arent black people like 10% of us population? You dont have ro look further

Re: AI models miss disease in Black and female patients

#143

Post by sapphicsnail »

balamatom wrote: post #121
timewizard wrote: post #86

Earlier quoted context omitted.

LLMs don't and cannot want things. Human beings also like it when the future is mostly like the past. They just call that "predictability." Human data is bias. You literally cannot remove one from the other. There are some people who want to erase humanity's will and replace it with an anthropomorphized algorithm. These people concern me.

The most concerning people are -- as ever -- those who only think that they are thinking. Those who keep trying to fit square pegs into triangular holes without, you know, stopping to reflect: who gave them those pegs in the first place, and to what end? Why be obtuse? There is no "anthropomorphic fallacy" here to dispel. You know very well that "LLMs want" is simply a way of speaking about teleology without antagoni…

Humans anthropocize all sorts of things but there are way bigger consequences for treating current AI like a human than someone anthropocizing their dog.

I know plenty of people that believe LLMs think and reason the same way as humans do and it leads them to make bad choices. I'm really careful about the language I use around such people because we understand expressions like, "the AI thought this" very differently.

Re: AI models miss disease in Black and female patients

#144

Post by bobthepanda »

encipriano wrote: post #139
kcorbitt wrote: post #118

Earlier quoted context omitted.

We may be in a simulation, but your odds of being alive to see this (conditioned on being born as a human at some point) aren't that low. Around 7% of all humans ever born are alive today!

I dont believe that percentage. Especially considering how spread the homo branch already was more than 100 000 years ago. And from which point do you start counting? Homo erectus?

I would imagine this is probably the source, which benchmarks using the last 200,000 years. https://www.prb.org/articles/how-many-people-have-ever-lived...

Given that we only hit the first billion people in 1804 and the second billion in 1927 it's not all that shocking.

Re: AI models miss disease in Black and female patients

#145

Post by thaumasiotes »

hnlmorg wrote: post #136
thaumasiotes wrote: post #134

Earlier quoted context omitted.

People seem to have started to use "LLM" to refer to any suite of software that includes an LLM somewhere within it; you can see them talking about LLM-generated art, for example.

Was it ascii art? ;)

https://hamatti.org/posts/art-forgery-llms-and-why-it-feels-...

People will just believe whatever they hear.

Re: AI models miss disease in Black and female patients

#146

Post by _l7dh »

orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

As Sara Hooker discussed in her paper https://www.cell.com/patterns/fulltext/S2666-3899(21)00061-1..., bias goes way beyond data.

Re: AI models miss disease in Black and female patients

#147

Post by jhanschoo »

orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

I like how the author used neo-Greek words to sneak in graphic imagery that would normally be taboo in this register of writing

Re: AI models miss disease in Black and female patients

#148

Post by XorNot »

encipriano wrote: post #139
kcorbitt wrote: post #118

Earlier quoted context omitted.

We may be in a simulation, but your odds of being alive to see this (conditioned on being born as a human at some point) aren't that low. Around 7% of all humans ever born are alive today!

I dont believe that percentage. Especially considering how spread the homo branch already was more than 100 000 years ago. And from which point do you start counting? Homo erectus?

That argument works both ways, it might be significantly higher depending how you count.

But this is also just the non-intuitiveness of exponential growth which has only now tapering off.

Re: AI models miss disease in Black and female patients

#149

Post by XorNot »

cratermoon wrote: post #119
mhuffman wrote: post #41

Earlier quoted context omitted.

"The model used in the new study, called CheXzero, was developed in 2022 by a team at Stanford University using a data set of almost 400,000 chest x-rays of people from Boston with conditions such as pulmonary edema, an accumulation of fluids in the lungs. Researchers fed their model the x-ray images without any of the associated radiologist reports, which contained information about diagnoses. " ... very interesting…

> Can it just tell which x-rays belong to Black or female patients and then use some latent racism or misogyny to change the diagnosis? The opposite. The dataset is for the standard model "white male", and the diagnoses generated pattern-matched on that. Because there's no gender or racial information, the model produced the statistically most likely result for white male, a result less likely to be correct for a pat…

The better question is just "are you actually just selecting for symptom occurrence by socioeconomic group?"

Like you could modify the question to ask "is the model better at diagnosing people who went to a certain school?" and simplistically the answer would likely seem to be yes.

Re: AI models miss disease in Black and female patients

#150

Post by echoangle »

elietoubi wrote: post #14

I came across a fascinating Microsoft research paper on MedFuzz ( https://www.microsoft.com/en-us/research/blog/medfuzz-explor... ) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaund…

> a sibling with alpha-thalassemia

I have no clue what that is or why it shouldn't change the diagnosis, but it seems to be a genetic thing. Is the problem that this has nothing to do with the described symptoms? Because surely, a sibling having a genetic disease would be relevant if the disease could be a cause of the symptoms?

Post reply on HN