Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep learning outperformed dermatologists in melanoma image classification task

ejcancer.com

Post reply on HN
Print view View original

41–50 of 94 posts

Re: Deep learning outperformed dermatologists in melanoma image classification task

#41

Post by sgt101 »

ModernMech wrote: post #18
yumraj wrote: post #3

I wonder if the results would be similar were the dermatologists to see the actual patient, in person, and then diagnose. And, then a photo was taken to be diagnosed by the CNN. In other words, while dermatologists may have been outperformed by deep learning in image classification, it is not evident if deep learning could do the same against dermatologist diagnosing in person. Also, not clear what the overall ratio…

Maybe? It's hard to say. I had a question about a mole once and what they did was take a photo of it with a special camera apparatus and they sent the image off to be diagnosed while I waited around. The doctor looked at it personally but it seems like the actual diagnosis was made by someone who never even saw me.

Same for me, the doctor I saw did say (after 3 seconds) "it's not a cancer but we'll take a picture anyway to be sure." I wonder if the same sort of thing would have happened if there was any real question in his mind?

Re: Deep learning outperformed dermatologists in melanoma image classification task

#43

Post by sgt101 »

mikehollinger wrote: post #8

I find it most interesting to use tools like this to augment - not replace diagnosticians and specialists. Of course the key here is that the training set is crucial to building a high quality model - which of course needs a set of specialists to give their consensus on the diagnosis of the patient based on the images. Presuming those folks can agree - the technology becomes a force multiplier for good. If they disag…

So long as the system is implemented in such as way as to stop it becoming a crutch or a default. If it's a tool in the flow then I think such things (not this one given the expert review information in this thread?) might be very valuable.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#44

Post by 0xab »

claytonjy wrote: post #34
0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

Why would you ever balance your test data? If 80/20 is the actual population distribution, the sample that forms your test set should conform to that. Balance all you want in train/validation sets, but never the test set. Not balancing and using ROC is a terrible combo, but the metric is the problem, not the lack of artificial balance.

I agree, they should do one or the other.

The imbalance is totally artificial and objectionable though. Where's the evidence that doctors see a 80/20 split in real life? If there is going to be an imbalance they should make it reflect the actual statistics of the task that the doctors perform not some artificial number. It doesn't even reflect the statistics of the dataset they started with (which is 90/10 unblanaced).

Admittedly, the correct analysis for when the data is unbalanced is more annoying and ROC curves are easier to interpret. That's why in something like ImageNet even though the training set is imbalanced, the test set is is balanced.

Comparisons against humans are also harder when the data is imbalanced in a way that reflects the training set, not the task. Humans don't know they are supposed to say "no" 80% of the time. That rewards the machine and that isn't easy to correct (you can correct what you think about the machine results with respect to a baseline, but not what biases the humans had).

Re: Deep learning outperformed dermatologists in melanoma image classification task

#45

Post by sgt101 »

JoshTko wrote: post #5

It's hard to imagine an narrow image classification task that humans will be able to beat NNs.

Considering that the ground truth comes from humans I would say that humans always outperform NN's and that results which show otherwise are demonstrating the limitations of the data set or testing process.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#46

Post by michaelhoffman »

plus wrote: post #37
0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

Since this is a journal focused on cancer and not machine learning, I can understand why the editors would see this paper as being worthy for for publication. Unfortunately, many of the readers will read the paper uncritically. If possible, you should write a critical response to this paper, focusing on its methodological flaws, and send it to the editors. It doesn't have to be long; critical response are usually a c…

How to Publish a Scientific Comment in 1 2 3 Easy Steps

http://frog.gatech.edu/Pubs/How-to-Publish-a-Scientific-Comm...

I agree that a formal comment is best although not necessarily easy. A comment on PubPeer is easier but it will probably only be seen by those with the PubPeer extension.

I do machine learning in computational biology and cancer. The issues described in the parent comment are known among experts. It’s too bad so many others don’t know or care.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#47

Post by perturbation »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

> * They measure the wrong things that reward the network. Because the dataset is imbalanced you can't use an ROC curve, sensitivity, or specificity. You need to use precision and recall and make a PR curve. This is machine learning and stats 101.

A̶F̶A̶I̶K̶,̶ ̶a̶ ̶R̶O̶C̶ ̶c̶u̶r̶v̶e̶ ̶c̶a̶n̶ ̶b̶e̶ ̶m̶i̶s̶l̶e̶a̶d̶i̶n̶g̶ ̶f̶o̶r̶ ̶a̶n̶ ̶i̶m̶b̶a̶l̶a̶n̶c̶e̶d̶ ̶d̶a̶t̶a̶s̶e̶t̶,̶ ̶b̶u̶t̶ ̶t̶h̶e̶ ̶A̶U̶C̶ ̶i̶s̶ ̶s̶t̶i̶l̶l̶ ̶o̶k̶a̶y̶ ̶f̶o̶r̶ ̶s̶e̶l̶e̶c̶t̶i̶n̶g̶ ̶m̶o̶d̶e̶l̶s̶.̶ Edit: This is incorrect, a PR curve + PR AUC should be used for model selection if imbalanced. I agree it would be really misleading if they (say) just reported accuracy (since the null classifier of always guess negative would give 80% overall accuracy). I̶ ̶t̶h̶o̶u̶g̶h̶t̶ ̶t̶h̶a̶t̶ ̶t̶h̶e̶ ̶A̶U̶C̶ ̶f̶o̶r̶ ̶R̶O̶C̶ ̶c̶u̶r̶v̶e̶ ̶s̶h̶o̶u̶l̶d̶ ̶s̶t̶i̶l̶l̶ ̶b̶e̶ ̶a̶ ̶v̶a̶l̶i̶d̶ ̶m̶e̶a̶s̶u̶r̶e̶ ̶s̶i̶n̶c̶e̶ ̶i̶t̶'̶s̶ ̶s̶h̶o̶w̶i̶n̶g̶ ̶h̶o̶w̶ ̶m̶u̶c̶h̶ ̶b̶e̶t̶t̶e̶r̶ ̶t̶h̶e̶ ̶m̶o̶d̶e̶l̶ ̶p̶e̶r̶f̶o̶r̶m̶s̶ ̶t̶h̶a̶n̶ ̶r̶a̶n̶d̶o̶m̶ ̶g̶u̶e̶s̶s̶i̶n̶g̶.̶

How do you usually handle imbalanced data? I've had some success with SMOTE or weighted loss for imbalanced datasets, but I'm embarrassed to say I've been using AUC with ROC curves as the default - if this gives inferior model selection than AUC with PR curve I'll have to start doing that instead.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#48

Post by [deleted] »

plus wrote: post #37
0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

Since this is a journal focused on cancer and not machine learning, I can understand why the editors would see this paper as being worthy for for publication. Unfortunately, many of the readers will read the paper uncritically. If possible, you should write a critical response to this paper, focusing on its methodological flaws, and send it to the editors. It doesn't have to be long; critical response are usually a c…

[deleted]

Re: Deep learning outperformed dermatologists in melanoma image classification task

#49

Post by savagedata »

leelin wrote: post #17

Maybe a dumb question from a non-medical guy: are medical images considered "stationary" from a stats viewpoint? That is, will medical images of diseases we diagnose in the next 20 years look a lot like the ones from the past 20 years, or is there a danger of over-fitting on an evolving data set? Could either the technology or the biology of the disease evolve? In a prior life I was a quant trader, and financial mark…

There are always potential issues when a machine learning algorithm is applied over time.

Example #1: Let's say that cancer rates are increasing over time and cameras are improving over time. You might end up with a weird artifact in your model that higher resolution images are more likely to indicate cancer.

Example #2: Let's say that cancer-detecting algorithms are widely successful and so someone makes an app that lets you upload images of skin and the app tells you the probability of you having cancer. Suddenly a model that was trained on suspicious lesions is being used on normal freckles that people uploaded for fun. You end up with a lot of false positives. Maybe you try to combat that by including images uploaded to the app (that you somehow obtain labels for). But now you have a model that predicts that photos taken in brightly lit medical offices are likely to be cancer and blurry images taken in bathroom mirrors are not cancer.

You could argue that Example #2 is more about the difference between training data and data to be scored, but the fact remains that outside of tightly controlled scenarios, the way data is collected nearly always changes in time and ends up affecting model performance in unexpected ways.

Post reply on HN