Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models miss disease in Black and female patients

science.org

Post reply on HN
Print view View original

121–130 of 256 posts

Re: AI models miss disease in Black and female patients

#121

Post by balamatom »

timewizard wrote: post #86
orr94 wrote: post #15

"AIs want the future to be like the past, and AIs make the future like the past. If the training data is full of human bias, then the predictions will also be full of human bias, and then the outcomes will be full of human bias, and when those outcomes are copraphagically fed back into the training data, you get new, highly concentrated human/machine bias.” https://pluralistic.net/2025/03/18/asbestos-in-the-walls/#go…

LLMs don't and cannot want things. Human beings also like it when the future is mostly like the past. They just call that "predictability." Human data is bias. You literally cannot remove one from the other. There are some people who want to erase humanity's will and replace it with an anthropomorphized algorithm. These people concern me.

The most concerning people are -- as ever -- those who only think that they are thinking. Those who keep trying to fit square pegs into triangular holes without, you know, stopping to reflect: who gave them those pegs in the first place, and to what end?

Why be obtuse? There is no "anthropomorphic fallacy" here to dispel. You know very well that "LLMs want" is simply a way of speaking about teleology without antagonizing people who are taught that they should be afraid of precise notions ("big words"). But accepting that bias can lead to some pretty funny conflations.

For example, humanity as a whole doesn't have this "will" you speak of any more than LLMs can "want"; will is an aspect of the consciousness of the individual. So you seem to be be uncritically anthropomorphizing social processes!

If we assume those to be chaotic, in that sense any sort of algorithm is slightly more anthropomorphic: at least it works towards a human-given and therefore human-comprehensible purpose -- on the other hand, whether there is some particular "destination of history" towards which humanity is moving, is a question that can only ever be speculated upon, but not definitively perceived.

Re: AI models miss disease in Black and female patients

#122

Post by nomel »

ericmcer wrote: post #115
pelorat wrote: post #107

Earlier quoted context omitted.

I think the model needs to be thought about human anatomy, not just fed a bunch of scans. It needs to understand what ribs and organs are.

I don't think LLMs can achieve "understanding" in that sense.

These aren't LLM. Most of the neat things in science, involving AI, aren't LLM. Next word prediction has extremely limited use with non-text data.

Re: AI models miss disease in Black and female patients

#123

Post by jsemrau »

appleorchard46 wrote: post #37
andsoitis wrote: post #31

Earlier quoted context omitted.

> Its easy to imagine a dataset not skewed towards black people. ~15% of the population in North America, probably less in Europe, and way less in Asia. What about Africa?

That's not where most of the data is coming from. If it was we'd be seeing the opposite effect, presumably.

I suppose that's the problem I have with that study. T

Re: AI models miss disease in Black and female patients

#124

Post by consteval »

nonethewiser wrote: post #54
lalaithion wrote: post #40

Earlier quoted context omitted.

Most trans people have undergone gender affirming medical care. A trans man who has had a hysterectomy and is on testosterone will have a very different medical baseline than a cis woman. A trans woman who has had an orchiectomy and is on estrogen will have a very different medical baseline than a cis man. It is literally throwing out relevant medical information to attempt to ignore this.

How is that in any way in conflict with what he said? You're just making an argument for more inputs. Biological sex, hormone levels, etc.

> hormone levels, etc.

Right… their gender they identify as.

So sex, and then also the gender they identify as.

You can’t hide behind an “etc”. Expand that out and the conclusion is you really do need to know who is trans and who is cisgender when doing treatment.

Re: AI models miss disease in Black and female patients

#125

Post by cratermoon »

kjkjadksj wrote: post #53

This isn’t an AI problem but a general medical field problem. It is a big issue with basically any population centric analysis where the people involved in the study don’t have a perfect subset of the worlds population to model human health; they have a couple hundred blood samples from patients at a Boise hospital over the past 10 years perhaps. And they validate this population against some other available cohort t…

It's a generative AI LLM hype issue because it follows the confidence game playbook. Feed someone correct ideas and answers that fit their biases until they trust you, then when the time is right, suggest things that fit their biases but give incorrect (and exploitative) results.

Re: AI models miss disease in Black and female patients

#126

Post by jl6 »

matthewmacleod wrote: post #93
nonethewiser wrote: post #85

Earlier quoted context omitted.

>Sex assigned at birth is in many situations important medical information Which is not gender identity. As a result of being trans there may be things like hormone levels that are different than what you'd expect based on biological sex, which is why I say hormone levels are important, but how you identify is in fact irrelevant.

Well, this is clearly wrong – it's obvious, for example, that gender identity could have a significant impact on mental health. Regardless of that, you seem to agree that: - Sex assigned at birth is important medical information - Information about gender affirming treatments is important medical information So I don't think there's much to worry about there.

The problem is that over the past few decades there has been substantial conflation of sex and gender, with many information systems replacing the former with the latter, rather than augmenting data collection with the latter.

Re: AI models miss disease in Black and female patients

#127

Post by hoaxminion »

elietoubi wrote: post #14

I came across a fascinating Microsoft research paper on MedFuzz ( https://www.microsoft.com/en-us/research/blog/medfuzz-explor... ) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaund…

[dead]

Re: AI models miss disease in Black and female patients

#128

Post by lesuorac »

KittenInABox wrote: post #113
dekhn wrote: post #108

Earlier quoted context omitted.

It doesn't seem surprising at all. Genetic history correlates with race, and genetic history correlates with body-level phenotypes; race also correlates with socioeconomic status which correlates with body-level phenotypes. They are of course fairly complex correlations with many confounding factors and uncontrolled variables. It has been controversial to discuss this and a lot of discussions about this end up in fla…

What is the body-level phenotype of a ribcage by race? I think what baffles me is that black people as a group are more genetically diverse than every other race put together so I have no idea how you would identify race by ribcage x-rays exclusively.

If you have 2 samples where one is highly concentrated around 5 and the other is dispersed more evenly between 0 and 10 then for any value of 5 you should guess Sample 1.

But anyways, the article links out to a paper [1] but unfortunately the paper tries to theorize things that would explain how and they don't find one (which may mean the AI is cheating imo not theirs).

[1]: https://www.thelancet.com/journals/landig/article/PIIS2589-7...

Re: AI models miss disease in Black and female patients

#129

Post by verisimi »

balamatom wrote: post #121
timewizard wrote: post #86

Earlier quoted context omitted.

LLMs don't and cannot want things. Human beings also like it when the future is mostly like the past. They just call that "predictability." Human data is bias. You literally cannot remove one from the other. There are some people who want to erase humanity's will and replace it with an anthropomorphized algorithm. These people concern me.

The most concerning people are -- as ever -- those who only think that they are thinking. Those who keep trying to fit square pegs into triangular holes without, you know, stopping to reflect: who gave them those pegs in the first place, and to what end? Why be obtuse? There is no "anthropomorphic fallacy" here to dispel. You know very well that "LLMs want" is simply a way of speaking about teleology without antagoni…

> If we assume those to be chaotic, in that sense any sort of algorithm is slightly more anthropomorphic: at least it works towards a human-given and therefore human-comprehensible purpose -- on the other hand, whether there is some particular "destination of history" towards which humanity is moving, is a question that can only ever be speculated upon, but not definitively perceived.

Do you not think that if you anthropomorphise things that aren't actually anthropic, that you then insert a bias towards those things? The bias will actually discriminate at the expense of people.

If that is so, the destination of history will inevitably be misanthropic.

Misplaced anthropomorphism is a genuine, present concern.

Post reply on HN