Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

International evaluation of an AI system for breast cancer screening

nature.com

Post reply on HN
Print view View original

11–20 of 28 posts

Re: International evaluation of an AI system for breast cancer screening

#11

Post by sanxiyn »

dang wrote: post #8

Related from 2 days ago: https://news.ycombinator.com/item?id=21917747 . This looks like different work though?

Yes, it is a different work. The current one published at Nature is from DeepMind.

It is interesting to note the differences. For example, DeepMind notes "In our reader study, all of the radiologists were eligible to interpret screening mammograms in the USA, but did not uniformly receive fellowship training in breast imaging." whereas DeepHealth notes "All readers were fellowship trained in breast imaging", so +1 to DeepHealth.

On the other hand, DeepMind says "Where data were available, readers were equipped with contextual information typically available in the clinical setting, including the patient’s age, breast cancer history, and previous screening mammograms." while DeepHealth says "Radiologists did not have any information about the patients (such as previous medical history, radiology reports, and other patient records)", so +1 to DeepMind. And so on. These differences make direct comparison between studies very difficult.

Re: International evaluation of an AI system for breast cancer screening

#12

Post by nl »

sanxiyn wrote: post #11
dang wrote: post #8

Related from 2 days ago: https://news.ycombinator.com/item?id=21917747 . This looks like different work though?

Yes, it is a different work. The current one published at Nature is from DeepMind. It is interesting to note the differences. For example, DeepMind notes "In our reader study, all of the radiologists were eligible to interpret screening mammograms in the USA, but did not uniformly receive fellowship training in breast imaging." whereas DeepHealth notes "All readers were fellowship trained in breast imaging", so +1 to…

This "+1" thing is damaging and incorrect.

Depending on the context the model ends up being used in something that appears good may not be. For example the fellowship training thing - these non-fellowship trained radiologists are doing this task now, so it is absolutely reasonable to assess against them to test real-world performance.

It would be interesting to see if the fellowship trained radiologists did actually perform better in all circumstances (in some fields the better trained radiologists end up not using their skills on as broad a range of patients, so their performance is actually worse one some subsets of data).

Re: International evaluation of an AI system for breast cancer screening

#13

Post by sanxiyn »

nl wrote: post #12
sanxiyn wrote: post #11

Earlier quoted context omitted.

Yes, it is a different work. The current one published at Nature is from DeepMind. It is interesting to note the differences. For example, DeepMind notes "In our reader study, all of the radiologists were eligible to interpret screening mammograms in the USA, but did not uniformly receive fellowship training in breast imaging." whereas DeepHealth notes "All readers were fellowship trained in breast imaging", so +1 to…

This "+1" thing is damaging and incorrect. Depending on the context the model ends up being used in something that appears good may not be. For example the fellowship training thing - these non-fellowship trained radiologists are doing this task now, so it is absolutely reasonable to assess against them to test real-world performance. It would be interesting to see if the fellowship trained radiologists did actually…

+1 was mostly to indicate whether you should upgrade or downgrade the reported result to be comparable with other studies. I didn't mean to imply whether it improves clinical relevancy.

Re: International evaluation of an AI system for breast cancer screening

#14

Post by xarope »

rvz wrote: post #3
stoicShell wrote: post #2

> Screening mammography aims to identify breast cancer at earlier stages of the disease, when treatment can be more successful. Despite the existence of screening programmes worldwide, the interpretation of mammograms is affected by high rates of false positives and false negatives. Here we present an artificial intelligence (AI) system that is capable of surpassing human experts in breast cancer prediction. [...] >…

Exactly this. This is where I see AI possibly going: To be a complimentary tool or second pair of eyes to speed up the work for the professionals rather than replacing them. I also see this research as a very positive step forward for using AI for good and especially bringing highly accurate results that can used as a aid for health professionals. However, given that this research used a deep learning (DL) based AI s…

Agreed. The ability of the someone/AI to explain their decision making process, is critical in determining whether such a decision has been adequately thought out or not. If a PhD must go through a viva, surely it is also incumbent on anybody pushing "AI" to also be able to "survive" such a viva. Otherwise, we might as well just go back to the days of reading entrails, flipping coins, etc. [edit: typo on viva]

Re: International evaluation of an AI system for breast cancer screening

#15

Post by nl »

sanxiyn wrote: post #13
nl wrote: post #12

Earlier quoted context omitted.

This "+1" thing is damaging and incorrect. Depending on the context the model ends up being used in something that appears good may not be. For example the fellowship training thing - these non-fellowship trained radiologists are doing this task now, so it is absolutely reasonable to assess against them to test real-world performance. It would be interesting to see if the fellowship trained radiologists did actually…

+1 was mostly to indicate whether you should upgrade or downgrade the reported result to be comparable with other studies. I didn't mean to imply whether it improves clinical relevancy.

Yeah that is fair.

Re: International evaluation of an AI system for breast cancer screening

#16

Post by sanxiyn »

rvz wrote: post #3
stoicShell wrote: post #2

> Screening mammography aims to identify breast cancer at earlier stages of the disease, when treatment can be more successful. Despite the existence of screening programmes worldwide, the interpretation of mammograms is affected by high rates of false positives and false negatives. Here we present an artificial intelligence (AI) system that is capable of surpassing human experts in breast cancer prediction. [...] >…

Exactly this. This is where I see AI possibly going: To be a complimentary tool or second pair of eyes to speed up the work for the professionals rather than replacing them. I also see this research as a very positive step forward for using AI for good and especially bringing highly accurate results that can used as a aid for health professionals. However, given that this research used a deep learning (DL) based AI s…

Note that the system does produce localization. "In addition to producing a classification decision for the entire case, the AI system was designed to highlight specific areas of suspicion for malignancy."

Re: International evaluation of an AI system for breast cancer screening

#17

Post by jcims »

stoicShell wrote: post #2

> Screening mammography aims to identify breast cancer at earlier stages of the disease, when treatment can be more successful. Despite the existence of screening programmes worldwide, the interpretation of mammograms is affected by high rates of false positives and false negatives. Here we present an artificial intelligence (AI) system that is capable of surpassing human experts in breast cancer prediction. [...] >…

The only issue is that humans don't seem to do well at jobs in which another agent is at least plausibly reliable. The Tesla autopilot is an example of that, we tend to disconnect pretty quickly.

Another thing I find interesting is that Google was able to train a neural network on retinas and can reliably distinguish sex based on retinal image alone...something opthamologists basically can't do. So not only are these systems approaching human capability in tasks we can do, they can do things we can't. As medical data becomes more freely flowing (presumably) over the next couple of decades, i think we'll find that 'AI' can become even more reliable.

Re: International evaluation of an AI system for breast cancer screening

#18

Post by sanxiyn »

jcims wrote: post #17
stoicShell wrote: post #2

> Screening mammography aims to identify breast cancer at earlier stages of the disease, when treatment can be more successful. Despite the existence of screening programmes worldwide, the interpretation of mammograms is affected by high rates of false positives and false negatives. Here we present an artificial intelligence (AI) system that is capable of surpassing human experts in breast cancer prediction. [...] >…

The only issue is that humans don't seem to do well at jobs in which another agent is at least plausibly reliable. The Tesla autopilot is an example of that, we tend to disconnect pretty quickly. Another thing I find interesting is that Google was able to train a neural network on retinas and can reliably distinguish sex based on retinal image alone...something opthamologists basically can't do. So not only are these…

I think machine's advantage can be summarized as "good at aggregating weak signals". Humans excel at analyzing complex signals, but basically can't use signals weaker than some point. Machines have no trouble with weak signals.

Re: International evaluation of an AI system for breast cancer screening

#19

Post by jcims »

sanxiyn wrote: post #18
jcims wrote: post #17

Earlier quoted context omitted.

The only issue is that humans don't seem to do well at jobs in which another agent is at least plausibly reliable. The Tesla autopilot is an example of that, we tend to disconnect pretty quickly. Another thing I find interesting is that Google was able to train a neural network on retinas and can reliably distinguish sex based on retinal image alone...something opthamologists basically can't do. So not only are these…

I think machine's advantage can be summarized as "good at aggregating weak signals". Humans excel at analyzing complex signals, but basically can't use signals weaker than some point. Machines have no trouble with weak signals.

Perfectly describes what I was struggling to put into words, thank you.
Post reply on HN