Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI misses nearly one-third of breast cancers, study finds

emjreviews.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: AI misses nearly one-third of breast cancers, study finds

#71

Post by directevolve »

_heimdall wrote: post #55
directevolve wrote: post #26

The point of this study is that it suggests that fully AI-automated mammography can currently deliver 70% sensitivity in detecting breast cancer using this model. It does not enable us to compare AI to unaided human performance. As this study did not include healthy controls, there is no false positive rate. The false positive rate is a crucial missing metric, since the vast majority of women do not have breast cance…

Its interesting to see this valid argument raised against this use of AI to identify breast cancer. The lack of control groups is one of the more common concerns raised related to vaccines as well, the argument lands like a lead balloon there.

Vaccine studies use a different experimental design known as “longitudinal,” meaning they follow people over time. This study did not do that. It’s still a valid design, just limited in what it tells us.

Re: AI misses nearly one-third of breast cancers, study finds

#73

Post by triceratops »

sfink wrote: post #22

The title bothers me. It suggests to me that "AI" is a single thing. If two guys are tested and turn out to be not that great at reading MRI images, should the headline be "Male radiologists miss nearly one-third of breast cancers"? If it said "AI something ", I'd be fine with it. It's a statement about that something, not about AI in general. Use it as an adjective (short for "AI-using" I guess?), not a noun.

There are more radiologists than AI models that read MRIs.

Re: AI misses nearly one-third of breast cancers, study finds

#74

Post by d0liver »

levocardia wrote: post #5

The original study: https://link.springer.com/article/10.1007/s11547-025-02161-1 It was retrospective-only, i.e. a case series on women who were known to have breast cancer, so there were zero false negatives and zero true negatives, because all patients in the study truly had cancer. The AI system used was a ConvNet used commercially circa 2021, which is when the data for this case series were collected.

> there were zero false negatives

Wouldn't this mean that AI identitied them all has having cancer?

Re: AI misses nearly one-third of breast cancers, study finds

#75

Post by cbarnes99 »

d0liver wrote: post #74
levocardia wrote: post #5

The original study: https://link.springer.com/article/10.1007/s11547-025-02161-1 It was retrospective-only, i.e. a case series on women who were known to have breast cancer, so there were zero false negatives and zero true negatives, because all patients in the study truly had cancer. The AI system used was a ConvNet used commercially circa 2021, which is when the data for this case series were collected.

> there were zero false negatives Wouldn't this mean that AI identitied them all has having cancer?

They did all have cancer

Re: AI misses nearly one-third of breast cancers, study finds

#76

Post by mattkrause »

vessenes wrote: post #62
directevolve wrote: post #26

The point of this study is that it suggests that fully AI-automated mammography can currently deliver 70% sensitivity in detecting breast cancer using this model. It does not enable us to compare AI to unaided human performance. As this study did not include healthy controls, there is no false positive rate. The false positive rate is a crucial missing metric, since the vast majority of women do not have breast cance…

I'm not a medical researcher, but I am a computer guy; I was struck by something very different in the papers - the abstracts at least refer to "AI CAD" as what they're testing - no software information, no versioning - on the CS side, this stuff is of paramount importance to make sure we know how the software performs. On the medical side, we need statistically significant tests that physicians can know and rely on…

The link is essentially a press release. The information you want is (sorta) in the actual paper it describes *.

"The images were analyzed using a commercially available AI-CAD system (Lunit INSIGHT MMG, version 1.1.7.0; Lunit Inc.), developed with deep convolutional neural networks and validated in multinational studies [1, 4]."

It's presumably a proprietary model, so you're not going to get a lot more information about it, but it's also one that's currently deployed in clinics, so...it's arguably a better comparison than a SOTA model some lab dumped on GitHub. I'd add that the post headline is also missing the point of the article: many of the missed cases can be detected with a different form of imaging. It's not really meant to be a model shoot-out style paper.

* Kim, J. Y., Kim, J. J., Lee, H. J., Hwangbo, L., Song, Y. S., Lee, J. W., Lee, N. K., Hong, S. B., & Kim, S. (2025). Added value of diffusion-weighted imaging in detecting breast cancer missed by artificial intelligence-based mammography. La Radiologia medica, 10.1007/s11547-025-02161-1. Advance online publication. https://doi.org/10.1007/s11547-025-02161

Re: AI misses nearly one-third of breast cancers, study finds

#77

Post by swisniewski »

philipp-gayret wrote: post #36
swisniewski wrote: post #27

The description from the summaries sound very flawed. 1. They only tested 2 Radiologists. And they compared it to one model. Thus the results don’t say anything about how Radiologists in general perform against AI in general. The most generous thing the study can say is that 2 Radiologists outperformed a particular model. 2. The Radiologists were only given one type of image, and then only for those patients that wer…

This article is about measuring how often an AI missed cancer by giving it data only where we know there was cancer. > Am I missing something? Yes. The article is not about AI performance vs human performance. > Humans are very capable at finding patterns (even if they don’t exist) when they want to find a pattern Ironic

The article has the headline "AI Misses Nearly One-Third of Breast Cancers, Study Finds".

It also has the following quotes:

1. "The results were striking: 127 cancers, 30.7% of all cases, were missed by the AI system"

2. "However, the researchers also tested a potential solution. Two radiologists reviewed only the diffusion-weighted imaging"

3. "Their findings offered reassurance: DWI alone identified the majority of cancers the AI had overlooked, detecting 83.5% of missed lesions for one radiologist and 79.5% for the other. The readers showed substantial agreement in their interpretations, suggesting the method is both reliable and reproducible."

So, if you are saying that the article is "not about AI performance vs human performance", that's not correct.

The article very clearly makes claims about the performance of AI vs the performance of doctors.

The study doesn't have the ability to state anything about the performance of doctors vs the performance of AI, because of the issues I mentioned. That was my point.

But the study can't state anything about the sensitivity of AI either because it doesn't compare the sensitivity of AI based mammography (XRay) analysis with that of human reviewed mammography. Instead it compares AI based mammography vs human based DWI when the humans knew the results were all true positives. It's both a different task ("diagnose" vs "find a pattern to verify an existing diagnosis") and different data (XRay vs MRI).

So, I don't think the claims from the article are valid in any way. And the study seems very flawed.

Also, attempting to measure sensitivity without also measuring specificity seems doubly flawed, because there are very big tradeoffs between the two.

Increasing sensitivity while also decreasing specificity can lead to unnecessary amputations. That's a very high cost. Also, apparently studies have show that high false positive rates for breast cancer can lead to increased cancer risks because they deter future screening.

Given that I don't have access to the actual study, I have to assume I am missing something. But I don't think it's what you think I'm missing.

Re: AI misses nearly one-third of breast cancers, study finds

#78

Post by kelseyfrog »

gcr wrote: post #61
directevolve wrote: post #26

The point of this study is that it suggests that fully AI-automated mammography can currently deliver 70% sensitivity in detecting breast cancer using this model. It does not enable us to compare AI to unaided human performance. As this study did not include healthy controls, there is no false positive rate. The false positive rate is a crucial missing metric, since the vast majority of women do not have breast cance…

What sensitivity / specificity are trained radiologists able to receive?

Great question. It prompted me to search for and find a history of sensitivity in mammography[1].

Their conclusion is that 39% is supported by evidence. Furthermore, there is a persistent erroneous belief that mammographic sensitivity is 90-95%.

1. https://pmc.ncbi.nlm.nih.gov/articles/PMC6640096/#R4

Re: AI misses nearly one-third of breast cancers, study finds

#79

Post by operatingthetan »

f1shy wrote: post #15
operatingthetan wrote: post #7

Earlier quoted context omitted.

>The AI system used was a ConvNet used commercially circa 2021, which is when the data for this case series were collected. Does this mean that newer AI systems would perform significantly differently?

Not at all. There is no implication, implicit or explicit, that anything in the world is better or worse. It is just a statement of fact.

>better or worse.

Please quote where I used either word.

Re: AI misses nearly one-third of breast cancers, study finds

#80

Post by _heimdall »

dgacmu wrote: post #58
_heimdall wrote: post #55

Earlier quoted context omitted.

Its interesting to see this valid argument raised against this use of AI to identify breast cancer. The lack of control groups is one of the more common concerns raised related to vaccines as well, the argument lands like a lead balloon there.

Because it's generally unethical to not give someone a treatment known already to be safe and effective. Studies of new vaccines where there is not an existing vaccine _do_ use placebo controls. Heck, my son got placebo during moderna's pediatric covid vaccine trial (to our frustration. grin.) Subsequent trials generally compare against the best known current treatment as the control instead. This study has no such c…

The covid vaccines were a whole different beast, though interesting case studies they were done under emergency authorization and didn't follow standard protocols.

Vaccine studies today almost always use a previously approved vaccine as the "control" group. That isn't a true control and if you walk back the chain of approvals you'd be hard pressed to find a starting point that did use proper control groups.

Anyway, my point here wasn't to directly debate vaccines themselves, only to point out that its interest to me as someone without a career in health to see the same effective argument used in two different scenarios with drastically different common responses.

Post reply on HN