Deep learning outperformed dermatologists in melanoma image classification task

Post by 0xab » Tue, Apr 30, 2019, 6:18 PM UTC

plus wrote: post #37

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

Since this is a journal focused on cancer and not machine learning, I can understand why the editors would see this paper as being worthy for for publication. Unfortunately, many of the readers will read the paper uncritically. If possible, you should write a critical response to this paper, focusing on its methodological flaws, and send it to the editors. It doesn't have to be long; critical response are usually a c…

Post by Florin_Andrei » Tue, Apr 30, 2019, 6:23 PM UTC

sgt101 wrote: post #36

feral wrote: post #30

Earlier quoted context omitted.

First off, if I'm reading correctly, it outperformed on its test set. This is different as it doesn't get to see that at any point before it's final. If the authors have done a diligent job here, that should be good evidence of it's accuracy. It's also encouraging to see they do multiple training runs, getting similar accuracy, and that their ROC is generally better than not just the average physician, but almost all…

Remember the test set is derived from the same source as the training set. This is not the case in the wild.

Post by mehrdadn » Tue, Apr 30, 2019, 6:43 PM UTC

0xab wrote: post #51

plus wrote: post #37

Earlier quoted context omitted.

Since this is a journal focused on cancer and not machine learning, I can understand why the editors would see this paper as being worthy for for publication. Unfortunately, many of the readers will read the paper uncritically. If possible, you should write a critical response to this paper, focusing on its methodological flaws, and send it to the editors. It doesn't have to be long; critical response are usually a c…

This is a huge problem throughout science, not just ML. As scientists, we're rewarded for publishing cool new things that work, not for pointing out things that don't or for pointing out flaws in existing papers. If the point is to get people to not read one bad paper, it's just a waste of my time. Most papers are false and a lot of them should never have passed review. If the authors actually wanted to do good ML re…

Post by PierredeFermat » Tue, Apr 30, 2019, 6:50 PM UTC

0xab wrote: post #51

plus wrote: post #37

Earlier quoted context omitted.

Since this is a journal focused on cancer and not machine learning, I can understand why the editors would see this paper as being worthy for for publication. Unfortunately, many of the readers will read the paper uncritically. If possible, you should write a critical response to this paper, focusing on its methodological flaws, and send it to the editors. It doesn't have to be long; critical response are usually a c…

This is a huge problem throughout science, not just ML. As scientists, we're rewarded for publishing cool new things that work, not for pointing out things that don't or for pointing out flaws in existing papers. If the point is to get people to not read one bad paper, it's just a waste of my time. Most papers are false and a lot of them should never have passed review. If the authors actually wanted to do good ML re…

Post by sgt101 » Tue, Apr 30, 2019, 6:57 PM UTC

feral wrote: post #32

assblaster wrote: post #9

Earlier quoted context omitted.

Machines will never replace dermatologists, machines will only make them more efficient.

If you make a dermatologist 5x more efficient, don't you replace 80% of them? Or even better, allow them to spend more time on the hardest cases. And allow people with no access to a dermatologist now, access to a machine almost as good?