Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep learning outperformed dermatologists in melanoma image classification task

ejcancer.com

Post reply on HN
Print view View original

31–40 of 94 posts

Re: Deep learning outperformed dermatologists in melanoma image classification task

#31

Post by 1wheel »

leelin wrote: post #17

Maybe a dumb question from a non-medical guy: are medical images considered "stationary" from a stats viewpoint? That is, will medical images of diseases we diagnose in the next 20 years look a lot like the ones from the past 20 years, or is there a danger of over-fitting on an evolving data set? Could either the technology or the biology of the disease evolve? In a prior life I was a quant trader, and financial mark…

Yup, this thread has a nice overview of ways performance on a validation set can overestimate clinical performance:

https://twitter.com/IAmSamFin/status/1122271463170564100

Another example of change over time:

> One difficulty in such a comparison is that Gleason grading standards have shifted over time, so that scores below six are now rarely assigned, and assigning a higher grade has become more common

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3775342/

Re: Deep learning outperformed dermatologists in melanoma image classification task

#32

Post by feral »

assblaster wrote: post #9
canada_dry wrote: post #4

This is a great example of where we need to get humans out of the equation when (if) a machine is conclusively proven to perform consistently better. It was justified (cost wise) to replace many human labourers on auto assembly since machines don't get tired, need breaks, have off days. It could certainly be argued it is even more important in the field of health care (reduce costs and improve outcomes) for all forms…

Machines will never replace dermatologists, machines will only make them more efficient.

If you make a dermatologist 5x more efficient, don't you replace 80% of them?

Or even better, allow them to spend more time on the hardest cases. And allow people with no access to a dermatologist now, access to a machine almost as good?

Re: Deep learning outperformed dermatologists in melanoma image classification task

#33

Post by arturadib »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

Agreed. See this paper for a reputable reference in this space: https://www.nature.com/articles/nature21056

Re: Deep learning outperformed dermatologists in melanoma image classification task

#34

Post by claytonjy »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

Why would you ever balance your test data? If 80/20 is the actual population distribution, the sample that forms your test set should conform to that. Balance all you want in train/validation sets, but never the test set.

Not balancing and using ROC is a terrible combo, but the metric is the problem, not the lack of artificial balance.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#35

Post by sgt101 »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

Thank you for posting this; I can see that this evaluation came very easily to you because of your experience and expertise but to me it shows how much knowledge is required to evaluate something like this. There really should be a protocol defined around this kind of study that encodes the criticisms that you make here (and others) and stops publication of this kind of thing in its tracks.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#36

Post by sgt101 »

feral wrote: post #30
hprotagonist wrote: post #6

As always, let's see how well it does in live images. This system outperformed dermatologists on its own validation set of 100 images, which I would encourage you to interpret as "heartening preliminary evidence" but not much more. Posting high scores on your validation set is only as informative as your val set is representative of the real world. 70% specificity, 84% sensitivity looks OK on paper (maybe -- as anoth…

First off, if I'm reading correctly, it outperformed on its test set. This is different as it doesn't get to see that at any point before it's final. If the authors have done a diligent job here, that should be good evidence of it's accuracy. It's also encouraging to see they do multiple training runs, getting similar accuracy, and that their ROC is generally better than not just the average physician, but almost all…

Remember the test set is derived from the same source as the training set. This is not the case in the wild.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#37

Post by plus »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

Since this is a journal focused on cancer and not machine learning, I can understand why the editors would see this paper as being worthy for for publication. Unfortunately, many of the readers will read the paper uncritically.

If possible, you should write a critical response to this paper, focusing on its methodological flaws, and send it to the editors. It doesn't have to be long; critical response are usually a couple pages at most. This is likely the most effective way of removing (or at the very least, heavily qualifying) bad science from research journals.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#38

Post by sgt101 »

learntoplay wrote: post #24
hprotagonist wrote: post #6

As always, let's see how well it does in live images. This system outperformed dermatologists on its own validation set of 100 images, which I would encourage you to interpret as "heartening preliminary evidence" but not much more. Posting high scores on your validation set is only as informative as your val set is representative of the real world. 70% specificity, 84% sensitivity looks OK on paper (maybe -- as anoth…

Isn’t DeepMind about to release a medical product that will do something very similar to this? Right now I wouldn’t doubt how well these systems can perform as compared to trained specialists that rely on their eyes even for reading test results.

I wonder if these products will have to go through proper trials like drugs do? If not, why not?

Re: Deep learning outperformed dermatologists in melanoma image classification task

#39

Post by michaelhoffman »

claytonjy wrote: post #34
0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

Why would you ever balance your test data? If 80/20 is the actual population distribution, the sample that forms your test set should conform to that. Balance all you want in train/validation sets, but never the test set. Not balancing and using ROC is a terrible combo, but the metric is the problem, not the lack of artificial balance.

80/20 is not the actual population distribution though.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#40

Post by ppod »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

> test data should be balanced or they should correct for this in the analysis.

Why should it be balanced? It should be the expected natural clinical class distribution, no? The humans have priors about this too. If anything, it should be more imbalanced, as I would guess (I would hope!) that less than 20% of scans are malignant.

Post reply on HN