Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep learning outperformed dermatologists in melanoma image classification task

ejcancer.com

Post reply on HN
Print view View original

81–90 of 94 posts

Re: Deep learning outperformed dermatologists in melanoma image classification task

#81

Post by ska »

leelin wrote: post #17

Maybe a dumb question from a non-medical guy: are medical images considered "stationary" from a stats viewpoint? That is, will medical images of diseases we diagnose in the next 20 years look a lot like the ones from the past 20 years, or is there a danger of over-fitting on an evolving data set? Could either the technology or the biology of the disease evolve? In a prior life I was a quant trader, and financial mark…

Stationarity isn't really the issue here, as you don't typically analyse this data as measurements from a single stochastic process.

However - you have hit on a very real problem. Imaging systems have got better over time, imaging quality even on the nominally same system can be different from different sites. Image coverage can change by both policy and system capabilities, etc.

It's worse the more sophisticated the imaging systems are. Consider MRI, which is perhaps better thought of as equipment to perform physics experiments than as an imaging device. In that case, nominally equivalent scans from different vendors (even different generation from the same vendor) can have significantly different characteristics. And there is a ton of processing going on, there is no such thing as "raw" data here - even the vendors themselves may no longer be able to really (or at least easily) characterize what is being done.

So yes, in any machine learning applied to these data sets, you have a very real risk of learning odd characteristics of the sample data and hurting your generalization.

Biology isn't as likely to be a problem I think, but biological response to changing treatment protocols, sure.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#82

Post by 1e-9 »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

I would say your criticism is way off base. I've developed and fielded ML-based medical devices and this looks like a reasonable study that suggests they have a system worthy of further testing. There's nothing wrong with using an ROC curve here and they document the experience of the doctors, so they weren't hiding that and around 60 or so doctors had greater than 5 years experience. Also, studies like this generally don't use only biopsy-proven negatives, since that would bias the negatives towards those that were suspicious enough to biopsy. Without knowing more details than what the paper provides, I cannot say the results are valid, but I also don't see any terrible errors after a quick scan. The main weakness is probably the fact that the test set came from the same image archive used for development. As a result, there can be all sorts of biases the CNN is using to inflate its performance unbeknownst to the developers. The best way to eliminate that concern is to use a test set gathered through a different data collection effort using different clinics, but that is expensive and time consuming and not something I would do initially. This looks like a good first step and I would encourage the developers to carry it further.

EDIT: I'll add that the ratio of positives to negatives in the training set is irrelevant and in no way invalidates the study. As far as testing goes, there is always a balance you must strike in a reader study involving doctors. Ideally, you would have the exact ratio a doctor would encounter in practice, but for a screening study, that is typically impractical as you would need a huge number of cases and doctor time is expensive. A ratio of 1 positive to 4 negatives is entirely reasonable, although the doctors (particularly the less experienced ones) will almost certainly have an elevated sensitivity and reduced specificity since they will know it is an enriched set, but this is reasonable for ROC comparison purposes as it mostly just selects a different point on the doctor's personal ROC curve. Note that some studies even tell the doctors beforehand what percentage of cases are positive.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#83

Post by arkades »

0xab wrote: post #44
claytonjy wrote: post #34

Earlier quoted context omitted.

Why would you ever balance your test data? If 80/20 is the actual population distribution, the sample that forms your test set should conform to that. Balance all you want in train/validation sets, but never the test set. Not balancing and using ROC is a terrible combo, but the metric is the problem, not the lack of artificial balance.

I agree, they should do one or the other. The imbalance is totally artificial and objectionable though. Where's the evidence that doctors see a 80/20 split in real life? If there is going to be an imbalance they should make it reflect the actual statistics of the task that the doctors perform not some artificial number. It doesn't even reflect the statistics of the dataset they started with (which is 90/10 unblanaced…

> Where's the evidence that doctors see a 80/20 split in real life?

Cause they definitely don’t. Even in a select subpopulation - say, people going to a derm for screening - you’d expect one melanoma per 620 persons screened (as per the SCREEN trial). Since most people have more than one mole for evaluation, and even those with melanoma will have multiple innocent moles... a mole count >50 triggers a referral for screening, though in more cautious docs, possibly as few as 25...

If you wanna be really generous and consider our hypothetical high risk group to have an average of 10 moles per person, that’s 6209:1, not 80:20.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#84

Post by avip »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

It's almost as if publishing the thing was more important for the authors than the scientific value of the content.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#85

Post by argonaut »

feral wrote: post #78
argonaut wrote: post #72

Earlier quoted context omitted.

> 10,000 images collected from leading clinical centers internationally Lets say the data was collected from 5 different clinical centers. One risk is basically that when you deploy the model, it only works at those clinical centers due to idiosyncrasies specific to those centers. Or suppose certain doctors were more likely to take melanoma images, and certain doctors were more likely to take non-melanoma images, and…

> These are just some ideas, there could be any number of confounding factors. There _could_ be. But when the source dataset was carefully gathered for a competition and the academics are saying "Broad and international participation in image contribution ensures that the dataset contains a representative clinically relevant sample", talking about multiple different equipment and labs, things look pretty promising. A…

Their validation set consisted of 210 positive images. The test set consisted of 20 positive images.

These are very small evaluation sets for deep learning. My point is the work is promising but should be viewed with healthy skepticism (by default).

I would really not read anything in particular into "Broad and international participation... ...sample." That's just a claim in a paper, it's not "the truth".

Re: Deep learning outperformed dermatologists in melanoma image classification task

#86

Post by psoy »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

Sensitivity and recall are two names for the same thing, Mr Stats 101 :)

Also, please explain the problem with using ROC here. The probabilistic interpretation of ROC's AUC is the probability of correctly ranking a random mixed pair (i.e. ranking the positive example higher than a negative one). How is that metric affected by the 80/20 split of the test data? Genuinely curios here...

Re: Deep learning outperformed dermatologists in melanoma image classification task

#87

Post by feral »

argonaut wrote: post #85
feral wrote: post #78

Earlier quoted context omitted.

> These are just some ideas, there could be any number of confounding factors. There _could_ be. But when the source dataset was carefully gathered for a competition and the academics are saying "Broad and international participation in image contribution ensures that the dataset contains a representative clinically relevant sample", talking about multiple different equipment and labs, things look pretty promising. A…

Their validation set consisted of 210 positive images. The test set consisted of 20 positive images. These are very small evaluation sets for deep learning. My point is the work is promising but should be viewed with healthy skepticism (by default). I would really not read anything in particular into "Broad and international participation... ...sample." That's just a claim in a paper, it's not "the truth".

> These are very small evaluation sets for deep learning.

Evaluation is a statistics question, and it doesn't matter that the deep learning model used is high capacity and needs a lot of training data.

There's nothing inherently wrong with validating a complex model on a small amount of data.

The paper has a section 4.2 that gives a statistical analysis. Granted, it'd be nicer if they had enough data to show statistically significant differences.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#88

Post by dontreact »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

There is nothing wrong with using ROC for imbalanced data. It is also perfectly reasonable to use an enriched dataset for a reader study, this is the standard practice.

Re: Deep learning outperformed dermatologists in melanoma image classification task

#90

Post by rcheu »

0xab wrote: post #28

I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…

I work as an ML engineer, some thoughts:

The train/test data being imbalanced in the same way does give the model an advantage, but I don't think that making the test set 50% would solve the issue completely either. Doctors have been "trained" on the true distribution, while which is not 50% (I'd guess that the true distribution is actually extremely unbalanced).

The model isn't simply learning to predict no 80% of the time, it is learning the distribution of the data with respect to the input features. For example, let's say that we have a simple model with only 3 binary features. It may learn that when features X_0, X_1 and X_2 are 1, the probability of cancer is 70%. This isn't a simple multiplication of the true probability by the upscaling factor though--it depends on the percent of negative samples with this feature vector and the percent of positive samples with this feature vector.

If we are to change the test set to be 50% positive and keep the same train distribution, the model no longer has the correct information about cancer rates with respect to feature distributions, but neither does the dermatologist. The specificity and sensitivity continue to not be interpretable as predicted specificity and sensitivity in the real world.

There is no issue with reporting specificity/sensitivity if they had used the true distribution of cases. Yes, the curves/AUCs will look better than the precision/recall rates, but they do not mis-represent what the doctors are interested in (what percent of people will be missed, and what percent of healthy people will be subjected to unnecessary procedures).

Anyways, the classifier doesn't actually seem to be that good, there's actually doctors that were better than the classifiers if you check the paper.

Post reply on HN