31–40 of 87 posts
Useful to show the failure rate for humans, and humans assisted by systems.
AI finds nearly 2/3rds of breast cancers!
Depending how the costs of AI detection vs doctor, that genuinely might be enough to shift the math and be a net positive. If it is cheap enough to test 10x the current tested population, which would have lower, but non-zero rates of breast cancer, then[0] AI would result in more cancer detected and therefore more aggregate lives saved. [0]presumptively
Based on my very superficial medical understanding, screening is already the cheap part. But every false-positive would lead to a doctor follow up at best and a biopsy at worst. Not to mention the significant psychological effects this has on a patient.
So I would counter that the potential increase of false-positive MRI scans could be enough to tip off the scale to make screening less useful
The description from the summaries sound very flawed. 1. They only tested 2 Radiologists. And they compared it to one model. Thus the results don’t say anything about how Radiologists in general perform against AI in general. The most generous thing the study can say is that 2 Radiologists outperformed a particular model. 2. The Radiologists were only given one type of image, and then only for those patients that wer…
The did NOT test radiologists. There were NO healthy controls. They evaluated AI false negative rate and used exclusively unblinded radiologists to grade the level of visibility and other features of the cancer. Utility of the study is to evaluate potential AI sensitivity if used for mass fully automated screenings using mammography data. But says NOTHING about the CRUCIAL false positive rate (no healthy controls) an…
Can you clarify?
I also hinted at the fact that I only had access to the posted summary and the original linked article, and not the study. So if there is data I am missing… please enlighten me.
The title bothers me. It suggests to me that "AI" is a single thing. If two guys are tested and turn out to be not that great at reading MRI images, should the headline be "Male radiologists miss nearly one-third of breast cancers"? If it said "AI something ", I'd be fine with it. It's a statement about that something, not about AI in general. Use it as an adjective (short for "AI-using" I guess?), not a noun.
I trust the meaning of this article is just that it requires hospitals to rethink their decision to substitute all doctors today.
The title bothers me. It suggests to me that "AI" is a single thing. If two guys are tested and turn out to be not that great at reading MRI images, should the headline be "Male radiologists miss nearly one-third of breast cancers"? If it said "AI something ", I'd be fine with it. It's a statement about that something, not about AI in general. Use it as an adjective (short for "AI-using" I guess?), not a noun.
They will directly write that "Radiologists miss nearly one-third of breast cancers." I trust the meaning of this article is just that it requires hospitals to rethink their decision to substitute all doctors today.
The description from the summaries sound very flawed. 1. They only tested 2 Radiologists. And they compared it to one model. Thus the results don’t say anything about how Radiologists in general perform against AI in general. The most generous thing the study can say is that 2 Radiologists outperformed a particular model. 2. The Radiologists were only given one type of image, and then only for those patients that wer…
> Am I missing something?
Yes. The article is not about AI performance vs human performance.
> Humans are very capable at finding patterns (even if they don’t exist) when they want to find a pattern
Ironic
Earlier quoted context omitted.
My personal opinion: AI should be still kept out of anything mission critical, in all stages, except for evaluation. There is other comment very correctly noting that this result is on 100% positive input. Same AI in “real life” would score probably much better eventually. But as you point out, if used as a confirmation tool, is definitely bad.
> Same AI in “real life” would score probably much better eventually Either I don't understand your reasoning or you are very much wrong. A "real life" dataset would contain real negatives too and the result would be equal if false positive rate was zero and strictly worse if the rate was any higher. One should expect the same AI to score significantly worse in a real life setting.
What I mean with “score” is having a relatively high accuracy.
Come let’s do the math: incidence of BC is 1 every 12, lets say. Now let’s say we have 12000 patients:
Acuracy = (TP + TN) / (TP + TN + FP + FN) = (1000 + 11000) / (1000 + 11000 + 300 + 0) = 12000 / 12300 =0.976 the test is 97.6% accurate… pretty impressive huh?
Tell me if I’m wrong. Is a know fact that you have to be careful when doctor speak of % accuracy.
Shouldn't A.I. not be used in a way that it only tries to assist? E.g. a doctor takes a look first and if (s)he can't find anything then A.I. is checking as well (or in parallel).
There's a roundup of such findings here, but they're a mixed bag: https://www.uxtigers.com/post/humans-negative-value I suspect you need careful process design to get better outcomes, and it's not one-size-fits-all.
Earlier quoted context omitted.
Given that every positive case needs to be verified by a doctor anyway because the patient has breast cancer, and every negative case has to be checked because it does a worse job than traditional methods... It only costs more.
Depends on the false positive rate. Hypothetically one can 'just' tune the model so false positives are low. This will increase false negatives but those are 'free' as they don't require follow ups. So long as the decrease in cost per real positive[0] goes down there's a benefit to be had. [0] accounting for false positives, screening costs for true negatives, etc. etc.
Increase in false negative rate significantly reduces survival rate and increases cost of treatment. We have huge multiplication factor here so decreasing false negative rate is the net positive option at relatively low rates.