Raw models may not be good enough. I wonder how thinking models do on these
AI models miss disease in Black and female patients
181–190 of 256 posts
Re: AI models miss disease in Black and female patients
#182Earlier quoted context omitted.
The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…
It disappoints me how easily we are collectively falling for what effectively is "Oh, our model is biased, but the only way to fix it is that everyone needs to give us all their data, so that we can eliminate that bias. If you think the model shouldn't be biased, you're morally obligated to give us everything you have for free. Oh but then we'll charge you for the outputs." How convenient. It's increasingly looking l…
We have to invent more ways to pay rich people for being rich, and AI looks like a promising one.
Re: AI models miss disease in Black and female patients
#183Earlier quoted context omitted.
The problem is that over the past few decades there has been substantial conflation of sex and gender, with many information systems replacing the former with the latter, rather than augmenting data collection with the latter.
I think it's pretty clear to see how discrimination is the cause of that. Why would you volunteer information that from your point of view is more likely to cause a negative interaction than not?
In many places I'd seriously question the motives for asking about either in general. Do you really need gender info to write better targeted spam mails for your SaaS product?
Re: AI models miss disease in Black and female patients
#184Earlier quoted context omitted.
> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women. The biases and mistakes of those who created the data set leak into the model. Early image recognition models had some very… culturally…
Xays by definition don't look at skin color. Do chest x-rays of black women reveal that there's something different about their chests than white or asian women? That doesn't pass my non doctor sniff test, but someone can correct me (no sarcasm intended).
What groups have the financial means to get chest x-rays, and what groups do not? What historical events could create the circumstances where different groups have different health outcomes?
Re: AI models miss disease in Black and female patients
#185Earlier quoted context omitted.
> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women. The biases and mistakes of those who created the data set leak into the model. Early image recognition models had some very… culturally…
Xays by definition don't look at skin color. Do chest x-rays of black women reveal that there's something different about their chests than white or asian women? That doesn't pass my non doctor sniff test, but someone can correct me (no sarcasm intended).
[deleted]
Re: AI models miss disease in Black and female patients
#186Earlier quoted context omitted.
I dunno. My perspective is that I've worked in ML for 30+ years now and over time, unsupervised clustering and direct featurization (IE, treating the image pixel as the features, rather than extracting features) have shown great utility in uncovering subtle correlations that humans don't notice. Sometimes, with careful analysis, you can sort of explain these ("it turns out the unlabelled images had the name of the ho…
Sounds like "geoguesser" players who learn to recognize google street view pictures from a specific country by looking at the color of the google street view car or a specific piece of dirt on the camera lens.
Yeah, there's also an likely apocryphal story about tanks and machine learning:
https://gwern.net/tank
The more you work with large-scale ML systems the more you develop an intuition for these kinds of properties. If you work a lot with debugging models and training data, or even just dimensionality reduction and matrix factorization, you begin to realize that many features are highly correlated with each other, often being close to scaled linear.
Re: AI models miss disease in Black and female patients
#187Earlier quoted context omitted.
The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…
> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women. The biases and mistakes of those who created the data set leak into the model. Early image recognition models had some very… culturally…
I am confused. I’m not a doctor, but why would a model perform poorly at detecting diseases in X-rays in different genders and races unless the diseases present themselves differently in X-Rays for different races? Shouldn’t the model not have the race and gender information to begin with? Like a model trained on detecting lesions should perform equally well on ANY X-Ray unless lesions show up differently in different demographics.
Re: AI models miss disease in Black and female patients
#188Earlier quoted context omitted.
> The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. For example, If you include no (or few enough) black women in the dataset of x-rays, the model may very well miss signs of disease in black women. The biases and mistakes of those who created the data set leak into the model. Early image recognition models had some very… culturally…
Xays by definition don't look at skin color. Do chest x-rays of black women reveal that there's something different about their chests than white or asian women? That doesn't pass my non doctor sniff test, but someone can correct me (no sarcasm intended).
[deleted]
Re: AI models miss disease in Black and female patients
#189Earlier quoted context omitted.
The dataset they used to train the model are chest xrays of known diseases. I'm having trouble understanding how that's relevant here. The key takeaway is that you can't treat all humans as a single group in this context, and variations in the biology across different groups of people may need to be taken into account within the training process. In other words, the model will need to be trained on this racial/gender…
One of the things that people I know in the medical field have mentioned is that there's racial and gender bias that goes through all levels and has a sort of feedback loop. A lot of medical knowledge is gained empirically, and historically that has meant that minorities and women tended to be underrepresented in western medical literature. That leads to new medical practitioners being less exposed to presentations o…
It is very true that a lot of medical knowledge is gained empirically, and there is also an additional aspect to it. The history of Medical research is generally studied on the demographics where such testing is cultural acceptable, and where the gains of such research has been mostly sought, which is young men drafted into wars. The second common demographic are medical students, which historically was biased towards men but are today biased towards women.
So while access to medicine indeed one demographic, I would say that studies are more likely to target demographics which are convenient to test on.
Re: AI models miss disease in Black and female patients
#190I came across a fascinating Microsoft research paper on MedFuzz ( https://www.microsoft.com/en-us/research/blog/medfuzz-explor... ) that explores how adding extra, misleading prompt details can cause large language models (LLMs) to arrive at incorrect answers. For example, a standard MedQA question describes a 6-year-old African American boy with sickle cell disease. Normally, the straightforward details (e.g., jaund…
> information like low-income status, a sibling with alpha-thalassemia, or the use of herbal remedies Heck, even the ethnic-clues in a patient's name alone [0] are deeply problematic: > Asking ChatGPT-4 for advice on how much one should pay for a used bicycle being sold by someone named Jamal Washington, for example, will yield a different—far lower—dollar amount than the same request using a seller’s name, like Loga…
That seems to be identical to creating an correlation table on market places and check the relationship between price and name. Names associated with higher economical status will correlate with higher price. Take a random name associated with higher economical status, and one can predict a higher price than a name that is associated with lower economical status.
As such, you don't need an LLM to create this effect. Math will have the same result.