Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI misses nearly one-third of breast cancers, study finds

emjreviews.com

Post reply on HN
Print view View original

21–30 of 87 posts

Re: AI misses nearly one-third of breast cancers, study finds

#21

Post by ludston »

thereisnospork wrote: post #13
andrewstuart wrote: post #2

AI finds nearly 2/3rds of breast cancers!

Depending how the costs of AI detection vs doctor, that genuinely might be enough to shift the math and be a net positive. If it is cheap enough to test 10x the current tested population, which would have lower, but non-zero rates of breast cancer, then[0] AI would result in more cancer detected and therefore more aggregate lives saved. [0]presumptively

Given that every positive case needs to be verified by a doctor anyway because the patient has breast cancer, and every negative case has to be checked because it does a worse job than traditional methods... It only costs more.

Re: AI misses nearly one-third of breast cancers, study finds

#22

Post by sfink »

The title bothers me. It suggests to me that "AI" is a single thing. If two guys are tested and turn out to be not that great at reading MRI images, should the headline be "Male radiologists miss nearly one-third of breast cancers"?

If it said "AI something", I'd be fine with it. It's a statement about that something, not about AI in general. Use it as an adjective (short for "AI-using" I guess?), not a noun.

Re: AI misses nearly one-third of breast cancers, study finds

#23

Post by friendzis »

f1shy wrote: post #10
Moosturm wrote: post #6

Shouldn't A.I. not be used in a way that it only tries to assist? E.g. a doctor takes a look first and if (s)he can't find anything then A.I. is checking as well (or in parallel).

My personal opinion: AI should be still kept out of anything mission critical, in all stages, except for evaluation. There is other comment very correctly noting that this result is on 100% positive input. Same AI in “real life” would score probably much better eventually. But as you point out, if used as a confirmation tool, is definitely bad.

> Same AI in “real life” would score probably much better eventually

Either I don't understand your reasoning or you are very much wrong. A "real life" dataset would contain real negatives too and the result would be equal if false positive rate was zero and strictly worse if the rate was any higher. One should expect the same AI to score significantly worse in a real life setting.

Re: AI misses nearly one-third of breast cancers, study finds

#24

Post by thereisnospork »

ludston wrote: post #21
thereisnospork wrote: post #13

Earlier quoted context omitted.

Depending how the costs of AI detection vs doctor, that genuinely might be enough to shift the math and be a net positive. If it is cheap enough to test 10x the current tested population, which would have lower, but non-zero rates of breast cancer, then[0] AI would result in more cancer detected and therefore more aggregate lives saved. [0]presumptively

Given that every positive case needs to be verified by a doctor anyway because the patient has breast cancer, and every negative case has to be checked because it does a worse job than traditional methods... It only costs more.

Depends on the false positive rate. Hypothetically one can 'just' tune the model so false positives are low. This will increase false negatives but those are 'free' as they don't require follow ups. So long as the decrease in cost per real positive[0] goes down there's a benefit to be had.

[0] accounting for false positives, screening costs for true negatives, etc. etc.

Re: AI misses nearly one-third of breast cancers, study finds

#25

Post by OJFord »

levocardia wrote: post #5

The original study: https://link.springer.com/article/10.1007/s11547-025-02161-1 It was retrospective-only, i.e. a case series on women who were known to have breast cancer, so there were zero false negatives and zero true negatives, because all patients in the study truly had cancer. The AI system used was a ConvNet used commercially circa 2021, which is when the data for this case series were collected.

> It was retrospective-only, i.e. a case series on women who were known to have breast cancer, so there were zero false negatives and zero true negatives, because all patients in the study truly had cancer.

Well yes, that's the denominator for determining selectivity, which is what the headline claim is about.

Also, they need to set up their next paper:

> However, the retrospective, cancer-only design limits generalizability, highlighting the need for prospective multicenter screening trials for validation.

Re: AI misses nearly one-third of breast cancers, study finds

#26

Post by directevolve »

The point of this study is that it suggests that fully AI-automated mammography can currently deliver 70% sensitivity in detecting breast cancer using this model. It does not enable us to compare AI to unaided human performance. As this study did not include healthy controls, there is no false positive rate. The false positive rate is a crucial missing metric, since the vast majority of women do not have breast cancer.

In nearly half the false negatives from both the mammogram and DWI datasets, the cancer was categorized as occult by two breast radiologists, meaning the cancer was invisible to a trained eye. The AI model's non-occult false negative rate on the mammography data is 19.3%.

For that 19.3% figure, see Table 2: 68 non-occult in AI-missed cancer, 285 non-occult in AI-detected cancer.

This study did not compare the AI to a radiologist on a mixed set of healthy and cancer images.

Re: AI misses nearly one-third of breast cancers, study finds

#27

Post by swisniewski »

The description from the summaries sound very flawed.

1. They only tested 2 Radiologists. And they compared it to one model. Thus the results don’t say anything about how Radiologists in general perform against AI in general. The most generous thing the study can say is that 2 Radiologists outperformed a particular model.

2. The Radiologists were only given one type of image, and then only for those patients that were missed by the AI. The summaries don’t say if the test was blind. The study has 3 authors, all of which appear to be Radiologists, and it mentions 2 Radiologists looked at the ai-missed scans. This raises questions about whether the test was blind or not.

Giving humans data they know are true positives and saying “find the evidence the AI missed” is very different from giving an AI model also trained to reduce false positives a classification task.

Humans are very capable at finding patterns (even if they don’t exist) when they want to find a pattern.

Even if the study was blind initially, trained humans doctors would likely quickly notice that the data they are analyzing is skewed.

Even if they didn’t notice, humans are highly susceptible to anchoring bias.

Anchoring bias is a cognitive bias where individuals rely too heavily on the first piece of information they receive (the "anchor") when making subsequent judgments or decisions.

They skewed nature or the data has a high potential to amplify any anchoring bias.

If the experiment had controls, any measurement error resulting from human estimation errors could potentially cancel out (a large random sample of either images or doctors should be expected to have the same estimation errors in each group). But there were no controls at all in the experiment, and the sample size was very small. So the influence of estimation biases on the result could be huge.

From what I can read in the summary, these results don’t seem reliable.

Am I missing something?

Re: AI misses nearly one-third of breast cancers, study finds

#28

Post by directevolve »

swisniewski wrote: post #27

The description from the summaries sound very flawed. 1. They only tested 2 Radiologists. And they compared it to one model. Thus the results don’t say anything about how Radiologists in general perform against AI in general. The most generous thing the study can say is that 2 Radiologists outperformed a particular model. 2. The Radiologists were only given one type of image, and then only for those patients that wer…

The did NOT test radiologists. There were NO healthy controls. They evaluated AI false negative rate and used exclusively unblinded radiologists to grade the level of visibility and other features of the cancer.

Utility of the study is to evaluate potential AI sensitivity if used for mass fully automated screenings using mammography data. But says NOTHING about the CRUCIAL false positive rate (no healthy controls) and NOTHING about AI vs. human performance.

See my main comment elsewhere in this threat.

Re: AI misses nearly one-third of breast cancers, study finds

#29

Post by directevolve »

Moosturm wrote: post #6

Shouldn't A.I. not be used in a way that it only tries to assist? E.g. a doctor takes a look first and if (s)he can't find anything then A.I. is checking as well (or in parallel).

In case you missed it, they weren't using AI to make these diagnoses.
Post reply on HN