Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI recognition of patient race in medical imaging: a modelling study

thelancet.com

Post reply on HN
Print view View original

111–120 of 180 posts

Re: AI recognition of patient race in medical imaging: a modelling study

#111

Post by towaway15463 »

cameldrv wrote: post #109
towaway15463 wrote: post #70

Earlier quoted context omitted.

Without knowing the actual outcome, isn’t there also a possibility of error due to not knowing the race of the individual? They used mammogram images in the study and it is well known that incidence of breast cancer varies by race. Removing that information from the model could result in worse performance.

Well one thing you wouldn’t want to do is take the output of this model and then apply a correction factor for race on top of it, because the model is already taking that into account.

Is that true or would it help as a tie breaker in cases where the confidence was just at or below the threshold?

Re: AI recognition of patient race in medical imaging: a modelling study

#112

Post by hugh-avherald »

inglor_cz wrote: post #110
chopin wrote: post #100

Earlier quoted context omitted.

I am surprised that this is a surprise. At least color vision is encoded in the X-Chromosome so there should be variation as males have only one which can be expressed.

It is a surprise, because the retina as an organ is very well visible and observable in living people, so we have a ton of observational data and practical clinical experience. But despite that, humans haven't noticed anything.

I've never seen a retina which had been separated from a human.

Re: AI recognition of patient race in medical imaging: a modelling study

#113

Post by amarshall »

codefreeordie wrote: post #107
amarshall wrote: post #105

Earlier quoted context omitted.

Maybe, maybe not. Hard to say—which is the problem they call out in the paper > efforts to control [model race-prediction] when it is undesirable will be challenging and demand further study

The correlation being "undesirable" to the individuals doing the research does not mean that the correlation is inaccurate. I mean, sure, there are tons of ways for garbage data to sneak into ML models -- though these guys tried pretty hard to control for that -- but if the model actually determined that "race" is a meaningful feature, then that might be because it is, and science should be concerned with what is, no…

If one believes and proclaims that they have controlled for variable X, but they haven’t actually done so, then their results and analysis may well be invalid or misleading because of that. Whether they actually should have controlled for X or not is orthogonal.

Re: AI recognition of patient race in medical imaging: a modelling study

#114

Post by ASalazarMX »

MontyCarloHall wrote: post #96
KaiserPro wrote: post #65

Earlier quoted context omitted.

> skeletal racial differences £10 says that its not that. Anatomy is extraordinarily hard, and AI isn't that good, yet. Sure different races have different layouts, but often that's only really obvious post mortem. (ie when you can yank out the bones and look at them, there are of course corner cases where high res CAT/MRI scans can pull out decent skeletal imagery in 3D) There are other cases, but that should be eas…

>I'd say its probably picking up on the style of imaging, rather than anything anatomical Certainly possible! They do control for hospital and machine … >Race prediction performance was also robust across models trained on single equipment and single hospital location on the chest x-ray and mammogram datasets … but it’s also possible that different chest x-rays were being used for different diagnostic purposes and th…

The weak factor in these AI studies seems to be data set normalization.

Re: AI recognition of patient race in medical imaging: a modelling study

#115

Post by ZeroGravitas »

hellohowareu wrote: post #5

Simply go to google image and search: "skeletal racial differences". subspecies are found across species-- they happen based on geographic dispersion and geographic isolation, which humans underwent for tens and hundreds of thousands of years. Welcome to the sciences of anatomy, anthropology, and forensics. other differences: - slow twitch vs fast twitch muscle - teeth shape - shapes and colors of various parts - gen…

The problem with 'race' as a concept isn't that you can genetically tell people apart.

Our tools are so precise you can tell which parent a set of cousins had with DNA tests, this doesn't make them a different species/sub-species or race from each other, even if one group has red hair and the other has black.

It's the pointless lumping together of people who are genetically distinct and drawing arbitrary, unscientific lines that's the issue.

Presumably the same experiments that can detect Asian Vs Black Vs White could also detect the entirely made up 'races' of Asian orBlack, AsianorWhite and WhiteorBlack since those are logically equivalent.

So are the races I made up a moment ago real things? No. But a computer can predict which category I'd assign, doesn't that make them real and important racial classifications? No it means my made up classifications map to other real genetic concepts at a lower level, like red hair.

Re: AI recognition of patient race in medical imaging: a modelling study

#116

Post by tedivm »

krona wrote: post #108
tedivm wrote: post #90

Earlier quoted context omitted.

>If I had to bet, and I knew where the data was coming from, I'd say its probably picking up on the style of imaging, rather than anything anatomical. Not all x-rays have bones in, and not all bones differ reliably to detect race. This was my guess as well. I've spent a lot of time around radiology and AI (I used to work at a company specializing in it) and we read a lot of the failure cases as well. There was one ex…

> one hospital was for higher risk patients- so it learned to assign all patients from that hospital to the disease category simply because they were at that hospital. That just sounds like poor feature selection/engineering. Garbage in, garbage out.

Yeah there are definitely ways they would have avoided that, but it's just one example of many. The whole point of ML is that it picks up on learned patterns. The problem is that it can be difficult to identify what it is learning from- this paper itself says they do not know what is causing it to make these predictions. As a result it is difficult to validate that the model is doing what people think it is.

Re: AI recognition of patient race in medical imaging: a modelling study

#117

Post by Ancapistani »

codefreeordie wrote: post #76
amarshall wrote: post #54

Earlier quoted context omitted.

Just because the model relies on race in some way doesn’t mean that we know it relies on it. I.e., the model is, unbeknownst to us, biased on race in inaccurate ways.

Presumably the model would actually be biased on race in accurate ways, if it found the correlation itself

I could be entirely wrong here, so if you've got more context in this area by all means correct me.

Consider an "AI" that rates the probability of recidivism for prisoners nearing their parole date. That score would then be presented to the parole board, and taken into consideration in determining whether or not to grant parole. If this AI were accidentally/incidentally accurately determining the race of the prisoner, then the output score would take that into account as well. Black men have a recidivism rate significantly higher than other groups[1]. The reasons for the above aside - it's a complex topic, and outside the scope of this analogy - this is extremely undesirable behavior for a process that is intended to remove human biases.

You might then ask, how does this relate to medical imaging? Medical decisions are regularly made based on the expected lifespan of the individual. It makes little sense to aggressively treat leukemia in a patient who is currently undergoing unrelated failure of multiple organs. Similarly it would likely make sense for a healthy 30-year-old to undergo a joint replacement and associated physical therapy, because that person can reasonably be expected to live for an additional 40 years while the same treatment wouldn't make sense for a 70-year-old with long-term chronic issues. This concept is commonly represented as "QALY" - "quality-adjusted life years".

Life expectancy can vary significantly based on race[2].

An AI that evaluates medical imagery that considers QALY in providing a care recommendation may result in a positive indicator for a white hispanic woman and a negative indicator for a black non-hispanic man, with all else being equal and with race as the only differentiator.

In short - it's not necessarily a bad thing for a model to be able to predict the race of the input imagery. The problem is that we don't know why it can do so. Unless we know that, we can't trust that the output is actually measuring what we intend it to be measuring.

1: https://prisoninsight.com/recidivism-the-ultimate-guide/ 2: https://www.cdc.gov/nchs/products/databriefs/db244.htm

Re: AI recognition of patient race in medical imaging: a modelling study

#118

Post by codefreeordie »

amarshall wrote: post #113
codefreeordie wrote: post #107

Earlier quoted context omitted.

The correlation being "undesirable" to the individuals doing the research does not mean that the correlation is inaccurate. I mean, sure, there are tons of ways for garbage data to sneak into ML models -- though these guys tried pretty hard to control for that -- but if the model actually determined that "race" is a meaningful feature, then that might be because it is, and science should be concerned with what is, no…

If one believes and proclaims that they have controlled for variable X, but they haven’t actually done so, then their results and analysis may well be invalid or misleading because of that. Whether they actually should have controlled for X or not is orthogonal.

Oh, yes, sorry. If by the correlation being possibly-undesirable you meant that it was possibly-spurious due to incompletely controlling for some bias in the source data, then yes, conclusions based on a model which found such a spurious correlation caused by incomplete input control might be undesirably biased in a not-accurate fashion.

This study appears to have done a good job controlling for known biases that could have been proxies for race, but it is presumably possible that they missed something and tainted the data

Re: AI recognition of patient race in medical imaging: a modelling study

#119

Post by codefreeordie »

Ancapistani wrote: post #117
codefreeordie wrote: post #76

Earlier quoted context omitted.

Presumably the model would actually be biased on race in accurate ways, if it found the correlation itself

I could be entirely wrong here, so if you've got more context in this area by all means correct me. Consider an "AI" that rates the probability of recidivism for prisoners nearing their parole date. That score would then be presented to the parole board, and taken into consideration in determining whether or not to grant parole. If this AI were accidentally/incidentally accurately determining the race of the prisoner…

At the risk of discussing sensitive topics on a platform ill-suited:

If, in your hypothetical recidivism case, an AI "accurately" determined that a pattern of higher recidivism-related features was correlated to race, and was able to determine "accurately" that the specific subset of recidivism-related features predicted race, why would it be wrong to make parole decisions using those recidivism-related features?

Re: AI recognition of patient race in medical imaging: a modelling study

#120

Post by dang »

The submitted title ("AI identifies race from xray, researchers don't know how") broke the site guidelines by editorializing. Submitters: please don't do that - it eventually causes your account to lose submission privileges.

From the guidelines (https://news.ycombinator.com/newsguidelines.html):

"Please use the original title, unless it is misleading or linkbait; don't editorialize."

Post reply on HN