Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

.98 AUC classifying race from chest x-rays (external validation)

lukeoakdenrayner.wordpress.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: .98 AUC classifying race from chest x-rays (external validation)

#11

Post by nightcracker »

pvitz wrote: post #4

I am maybe totally naive, but if this feature is bad, then why do all of these models even include race in the training set? Removing it would also remove the potential bias? How is self-identified race potentially relevant for disease identification? Also, I guess humans don't perform well in this task because nobody is studying x-rays for the purpose of identifying race?

The point is that if a neural network can predict the relationship when labelled with such accuracy it means this information is unavoidably encoded within the image. The result is that even when race is not labeled that the AI might still be racist by automatically (and unknowingly) learning the race.

Re: .98 AUC classifying race from chest x-rays (external validation)

#12

Post by lukeor »

Agingcoder wrote: post #10
mlnewb2 wrote: post #9

Earlier quoted context omitted.

why is this a contradiction? If the radiologist doesn't see the patient, they can't appreciate when the AI might be making decisions based on race rather than clinically relevant info.

I understand that ai training is done without injecting race bias in the model, since radiologists don't know about it. Since the nn is supposed to generalize the training set, I don't understand how it can suddenly become racist - in other words, it may learn race, but I don't understand why it would be biased towards one ethnic group or another. Is my understanding correct? Edit: do you assume that the ai output, i…

Hi, author here. I'll try to answer here with two points.

1) race is not provided explicitly or intentionally during training, but medical practice is biased so it is reasonable to assume there is a signal in the data.

2) we know there must be a signal, since AI models learn it. The optimisation process should only discover features that correlate with the labels, but we see the ability to predict race in models trained to look for diseases like pneumonia (which do not appear different for people from different racial groups).

Re: .98 AUC classifying race from chest x-rays (external validation)

#13

Post by Agingcoder »

lukeor wrote: post #12
Agingcoder wrote: post #10

Earlier quoted context omitted.

I understand that ai training is done without injecting race bias in the model, since radiologists don't know about it. Since the nn is supposed to generalize the training set, I don't understand how it can suddenly become racist - in other words, it may learn race, but I don't understand why it would be biased towards one ethnic group or another. Is my understanding correct? Edit: do you assume that the ai output, i…

Hi, author here. I'll try to answer here with two points. 1) race is not provided explicitly or intentionally during training, but medical practice is biased so it is reasonable to assume there is a signal in the data. 2) we know there must be a signal, since AI models learn it. The optimisation process should only discover features that correlate with the labels, but we see the ability to predict race in models trai…

Thanks a lot for taking the time to reply. I have to admit that I have found your post unexpectedly super interesting. It's quite subtle, and while I can see all the facts, there are a few logical steps which don't quite make sense to me.

So you're saying that 1. Unlike what the blog post says, there is practice bias even in radiology. This would indeed explain why the model can learn 'racial bias'.

2. This is less clear to me. Just like humans can't see race on radio images, humans might be unable to see differences for diseases like pneumonia, but the nn could see them, no? In other words, how do you know that the differences have to come from hidden racial bias, and not from hidden pathological differences (that you don't know about, just like you've just discovered hidden racial bias) ?

Re: .98 AUC classifying race from chest x-rays (external validation)

#14

Post by Agingcoder »

Agingcoder wrote: post #13
lukeor wrote: post #12

Earlier quoted context omitted.

Hi, author here. I'll try to answer here with two points. 1) race is not provided explicitly or intentionally during training, but medical practice is biased so it is reasonable to assume there is a signal in the data. 2) we know there must be a signal, since AI models learn it. The optimisation process should only discover features that correlate with the labels, but we see the ability to predict race in models trai…

Thanks a lot for taking the time to reply. I have to admit that I have found your post unexpectedly super interesting. It's quite subtle, and while I can see all the facts, there are a few logical steps which don't quite make sense to me. So you're saying that 1. Unlike what the blog post says, there is practice bias even in radiology. This would indeed explain why the model can learn 'racial bias'. 2. This is less c…

(replying to myself about 2) You say the race can't be seen in the image by humans because it's not here, and therefore the race information must come from bias introduced by the radiologist.

I guess it depends on whether all differences are visible to the human eye (and there's nothing pathological hidden that the nn 'sees') , and whether you can prove (a very hard thing in stats) that there's no way you can possibly extract race in a different manner.

Now, assuming the data is racially biased because of medical practice biased, forgive me for being naive, but, why is this surprising / major?

Isn't this just another instance of models being trained on bad data, and there are already plenty of examples in ia ?

Edit:.. Unless the actual finding becomes 'radiology is (unexpectedly) racially biased, so much so that an ia can learn it'?

Re: .98 AUC classifying race from chest x-rays (external validation)

#15

Post by lukeor »

Agingcoder wrote: post #14
Agingcoder wrote: post #13

Earlier quoted context omitted.

Thanks a lot for taking the time to reply. I have to admit that I have found your post unexpectedly super interesting. It's quite subtle, and while I can see all the facts, there are a few logical steps which don't quite make sense to me. So you're saying that 1. Unlike what the blog post says, there is practice bias even in radiology. This would indeed explain why the model can learn 'racial bias'. 2. This is less c…

(replying to myself about 2) You say the race can't be seen in the image by humans because it's not here, and therefore the race information must come from bias introduced by the radiologist. I guess it depends on whether all differences are visible to the human eye (and there's nothing pathological hidden that the nn 'sees') , and whether you can prove (a very hard thing in stats) that there's no way you can possibl…

I never said radiology didn't produce biased results, just that we don't know the race of our patients most of the time.

There are lots of ways bias can still occur, like in who gets referred for scans, when they get referred, what the referrer writes on the request form, how the technologist takes the images (I could tell you some horribly racist stories about a few ultrasonographers I've worked with), and so on.

And all of this is based on previous work that AI produces bias (when trained on these datasets). If it was useful differences that drove AI learning about race, the models would not produce disparities. We went looking for how it is interacting with race because we already knew it was producing unacceptable outcomes. The big news here is that it is so easy to learn race that this effect is almost certainly not isolated to the systems tested so far.

Re: .98 AUC classifying race from chest x-rays (external validation)

#16

Post by Agingcoder »

lukeor wrote: post #15
Agingcoder wrote: post #14

Earlier quoted context omitted.

(replying to myself about 2) You say the race can't be seen in the image by humans because it's not here, and therefore the race information must come from bias introduced by the radiologist. I guess it depends on whether all differences are visible to the human eye (and there's nothing pathological hidden that the nn 'sees') , and whether you can prove (a very hard thing in stats) that there's no way you can possibl…

I never said radiology didn't produce biased results, just that we don't know the race of our patients most of the time. There are lots of ways bias can still occur, like in who gets referred for scans, when they get referred, what the referrer writes on the request form, how the technologist takes the images (I could tell you some horribly racist stories about a few ultrasonographers I've worked with), and so on. An…

Thanks!

Re: .98 AUC classifying race from chest x-rays (external validation)

#17

Post by pvitz »

nightcracker wrote: post #11
pvitz wrote: post #4

I am maybe totally naive, but if this feature is bad, then why do all of these models even include race in the training set? Removing it would also remove the potential bias? How is self-identified race potentially relevant for disease identification? Also, I guess humans don't perform well in this task because nobody is studying x-rays for the purpose of identifying race?

The point is that if a neural network can predict the relationship when labelled with such accuracy it means this information is unavoidably encoded within the image. The result is that even when race is not labeled that the AI might still be racist by automatically (and unknowingly) learning the race.

If one only trains the model to correctly identify a disease (e.g. maximising the correct identification and minimising the error), then there is no space for the model to be "racist". I don't question that there is a signal inside the images, but someone has to change the model and training to then maliciously use that information.
Post reply on HN