The moving of the goal posts is a lot here. Measuring sensitivity only is still a useful thing and at least aids radiologists in their decision to use this model specifically and how to base their reliance on it. Also, why does every study have to compare all humans across human history to some particular model?
AI misses nearly one-third of breast cancers, study finds
41–50 of 87 posts
Re: AI misses nearly one-third of breast cancers, study finds
#42This seems a bit like a needlessly publicized finding. Surely our baseline assumption is that there are lots of systems that aren't very good at finding cancer. We're interested in findings that are good. You only need 1 good system to adopt. Yes, it's good scientific hygiene to do the study and publish it going "Well, this particular thing isn't good let's move on". But my expectation is you just going until you design a system that does do well and then adopt that system.
If I pluck a guy off the street, get him to analyze a load of MRI scans and he doesn't correctly identify cancer from them I'm not going to publish an article saying "Humans miss X% of breast cancers" am I.
Re: AI misses nearly one-third of breast cancers, study finds
#43This seems a bit like a needlessly publicized finding. Surely our baseline assumption is that there are lots of systems that aren't very good at finding cancer. We're interested in findings that are good. You only need 1 good system to adopt. Yes, it's good scientific hygiene to do the study and publish it going "Well, this particular thing isn't good let's move on". But my expectation is you just going until you des…
Many people are confused and think the Bitter Lesson is that you can just feed up a bigger and bigger model and eventually it becomes omnipotent.
Re: AI misses nearly one-third of breast cancers, study finds
#44Earlier quoted context omitted.
The did NOT test radiologists. There were NO healthy controls. They evaluated AI false negative rate and used exclusively unblinded radiologists to grade the level of visibility and other features of the cancer. Utility of the study is to evaluate potential AI sensitivity if used for mass fully automated screenings using mammography data. But says NOTHING about the CRUCIAL false positive rate (no healthy controls) an…
Huh? I was commenting that there were no controls and the doctors were given skewed data, so any conclusions of ai ability vs Dr ability seem misplaced. Which seems to be what you just said… so I am confused about what I said that was inaccurate. Can you clarify? I also hinted at the fact that I only had access to the posted summary and the original linked article, and not the study. So if there is data I am missing……
I was just reinforcing that point as your comment was worded in a way that left room for doubt. Sorry if this came across as critical toward you or implying you held a different interpretation.
Re: AI misses nearly one-third of breast cancers, study finds
#45So basically AI kills people.
This is Skynet 2.0 or 3.0. But shit. James Cameron may have to redo The Terminator, to include AI. Then again, who would watch such a movie?
Re: AI misses nearly one-third of breast cancers, study finds
#46This seems a bit like a needlessly publicized finding. Surely our baseline assumption is that there are lots of systems that aren't very good at finding cancer. We're interested in findings that are good. You only need 1 good system to adopt. Yes, it's good scientific hygiene to do the study and publish it going "Well, this particular thing isn't good let's move on". But my expectation is you just going until you des…
Many people are confused and think the Bitter Lesson is that you can just feed up a bigger and bigger model and eventually it becomes omnipotent.
They promised us that AI was The Solution. Now they have to deliver.
If the TechBros fail us here, we may then assume they may fail us everywhere else as well.
Re: AI misses nearly one-third of breast cancers, study finds
#47This seems a bit like a needlessly publicized finding. Surely our baseline assumption is that there are lots of systems that aren't very good at finding cancer. We're interested in findings that are good. You only need 1 good system to adopt. Yes, it's good scientific hygiene to do the study and publish it going "Well, this particular thing isn't good let's move on". But my expectation is you just going until you des…
I think finding that AI or at least specific model sold to be able to do something can't reasonable do it is entirely reasonable thing to publish.
In the end it is on the model marketer to prove that what they sell can do what it says. And counter examples is fully valid thing to then release.
Re: AI misses nearly one-third of breast cancers, study finds
#48Not all AIs are created equal.
Re: AI misses nearly one-third of breast cancers, study finds
#49"AI" doesn't exist. There are probably hundreds of different breast cancer detection algorithms. Maybe the SOTA isn't good enough yet. That doesn't mean AI in general is fundamentally incapable of correctly detecting it.
It is a useful umbrella term for all the systems in question.
Re: AI misses nearly one-third of breast cancers, study finds
#50As someone else has pointed out, I would like to know how this compares to humans.
I hope something good comes out of this, as I have known women whose lives were deeply affected by this.