61–70 of 94 posts
In the future your doctor will have an image scanner in their office and you'll get 'really cheap' diagnosis of this to back up the doctors opinion.
Then you'll go for biopsy etc..
This is a great example of where we need to get humans out of the equation when (if) a machine is conclusively proven to perform consistently better. It was justified (cost wise) to replace many human labourers on auto assembly since machines don't get tired, need breaks, have off days. It could certainly be argued it is even more important in the field of health care (reduce costs and improve outcomes) for all forms…
Machines will never replace dermatologists, machines will only make them more efficient.
Imaging specialties might end up becoming more "efficient" (such as radiology), but medical patient data which is used to diagnose and triage is some of the worst there is (source: I worked with it). The only positive outcome I see from ML for the medical field for the foreseeable future might be to reduce the number of misdiagnoses if all things go very well.
I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…
Why would you ever balance your test data? If 80/20 is the actual population distribution, the sample that forms your test set should conform to that. Balance all you want in train/validation sets, but never the test set. Not balancing and using ROC is a terrible combo, but the metric is the problem, not the lack of artificial balance.
Earlier quoted context omitted.
Since this is a journal focused on cancer and not machine learning, I can understand why the editors would see this paper as being worthy for for publication. Unfortunately, many of the readers will read the paper uncritically. If possible, you should write a critical response to this paper, focusing on its methodological flaws, and send it to the editors. It doesn't have to be long; critical response are usually a c…
How to Publish a Scientific Comment in 1 2 3 Easy Steps http://frog.gatech.edu/Pubs/How-to-Publish-a-Scientific-Comm... I agree that a formal comment is best although not necessarily easy. A comment on PubPeer is easier but it will probably only be seen by those with the PubPeer extension. I do machine learning in computational biology and cancer. The issues described in the parent comment are known among experts. It…
I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…
My understanding is the PR curve is preferable to ROC since the ROC can make it difficult to discern differences between models on imbalanced data; but the ROC is still a valid way to compare/measure models.
I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…
I wish they added this context in the limitations section. The paper only says:
"There are some limitations to this system. It remains an open question whether the design of the questionnaire had any influence on the performance of the dermatologists compared with clinical settings. Furthermore, clinical encounters with actual patients provide more information than that can be provided by images alone. Hänßle et al. showed that additional clinical data improve the sensitivity and specificity of dermatologists slightly [5]. Machine learning techniques can also include this information in their decisions. However, even with this slight improvement, the CNN would still outperform the dermatologists."
Your points hit on validity issues. Where would it fit on the errors of omission/commission scale?
It's hard to imagine an narrow image classification task that humans will be able to beat NNs.
Considering that the ground truth comes from humans I would say that humans always outperform NN's and that results which show otherwise are demonstrating the limitations of the data set or testing process.
Earlier quoted context omitted.
Considering that the ground truth comes from humans I would say that humans always outperform NN's and that results which show otherwise are demonstrating the limitations of the data set or testing process.
It's been proven that using humans as ground truth you can ultimately build a NN off that data which outperforms the humans.
Earlier quoted context omitted.
Yeah, this is the ML equivalent of "it works in vitro".
That's a lovely way of putting it. You're exactly right.
"This challenge leveraged a database of dermoscopic skin images from the ISIC Data Archive1 , which at the time of this publication contains over 10,000 images collected from leading clinical centers internationally, acquired from a variety of devices used at each center. The images are screened for both privacy and quality assurance. The associated clinical metadata has been vetted by recognized melanoma experts. Broad and international participation in image contribution ensures that the dataset contains a representative clinically relevant sam- ple". [Gutman et al.]
This paper selects a subset relevant to a certain condition (beyond my expertise).
But that sounds pretty good to me. If the test set in the paper is carefully randomly selected, as the authors described (and I've no reason to disbelieve them) then performance on test set should be a good proxy to actual performance on unseen data, as the underlying dataset is designed to be representative of clinical practice.
Of course that's not the same as having a clinically useful product.
But commenters here are knocking these cancer researchers as if they are idiots, and saying very harsh things about their methods, when it seems like 3 minutes reading their Dataset section gives reasons to think their setup is actually pretty ok?