71–80 of 94 posts
For humans and AI.
Why don't I have a record of my spots from being a baby?
Privacy left to implementing, but there's a few ways you could go.
Earlier quoted context omitted.
That's a lovely way of putting it. You're exactly right.
Look, I'm not a domain expert in the medical side of this, but the paper says they used a dataset, described in it's referenced paper as: "This challenge leveraged a database of dermoscopic skin images from the ISIC Data Archive1 , which at the time of this publication contains over 10,000 images collected from leading clinical centers internationally, acquired from a variety of devices used at each center. The image…
Lets say the data was collected from 5 different clinical centers. One risk is basically that when you deploy the model, it only works at those clinical centers due to idiosyncrasies specific to those centers. Or suppose certain doctors were more likely to take melanoma images, and certain doctors were more likely to take non-melanoma images, and both sets of doctors used different techniques to take images. These are just some ideas, there could be any number of confounding factors.
Basically, one should be default suspicious of most research papers working with small datasets (the test set is only 100 images - this is very small) that have not been deployed in the real world or otherwise validated independently. The root comment of this thread is basically saying this (a different comment is the one calling the researchers idiots).
Earlier quoted context omitted.
> Most papers are false and a lot of them should never have passed review. Do you mean this literally or is this a metaphor to illustrate the point? If you actually mean most papers are false it'd be nice to see a link on that!
John Ioannidis claims that "most published research is false" based on some rather dubious assumptions. https://www.annualreviews.org/doi/abs/10.1146/annurev-statis...
> For example, Roberto Novoa, a clinical dermatologist at Stanford University in the US, has described a time when he and his colleagues designed an algorithm to recognize skin cancer – only to discover that they’d accidentally designed a ruler detector instead, because the largest tumours had been photographed with rulers next to them for scale.
Source: https://physicsworld.com/a/neural-networks-explained/
Earlier quoted context omitted.
It's been proven that using humans as ground truth you can ultimately build a NN off that data which outperforms the humans.
What is meant by proven? What is meant by "the humans"? And how can any human say that is so?
Earlier quoted context omitted.
Why would you ever balance your test data? If 80/20 is the actual population distribution, the sample that forms your test set should conform to that. Balance all you want in train/validation sets, but never the test set. Not balancing and using ROC is a terrible combo, but the metric is the problem, not the lack of artificial balance.
I agree, they should do one or the other. The imbalance is totally artificial and objectionable though. Where's the evidence that doctors see a 80/20 split in real life? If there is going to be an imbalance they should make it reflect the actual statistics of the task that the doctors perform not some artificial number. It doesn't even reflect the statistics of the dataset they started with (which is 90/10 unblanaced…
This is a great example of where we need to get humans out of the equation when (if) a machine is conclusively proven to perform consistently better. It was justified (cost wise) to replace many human labourers on auto assembly since machines don't get tired, need breaks, have off days. It could certainly be argued it is even more important in the field of health care (reduce costs and improve outcomes) for all forms…
> This is a great example of where we need to get humans out of the equation now that a machine performs consistently better. No, it's not. The system is an image classifier with a HUGE false positive rate (and false negative rate). When false positive rates exceed actual incident rates in the population (or far exceed by orders of magnitude, in this case), then it's practically worthless. This is something that the…
When booking an appointment with your doctor, you could take a photo and submit it. If the NN detects something abnormal, it could be useful in prioritizing your appointment or at least flagging it for reviewal by the MD.
Earlier quoted context omitted.
Look, I'm not a domain expert in the medical side of this, but the paper says they used a dataset, described in it's referenced paper as: "This challenge leveraged a database of dermoscopic skin images from the ISIC Data Archive1 , which at the time of this publication contains over 10,000 images collected from leading clinical centers internationally, acquired from a variety of devices used at each center. The image…
> 10,000 images collected from leading clinical centers internationally Lets say the data was collected from 5 different clinical centers. One risk is basically that when you deploy the model, it only works at those clinical centers due to idiosyncrasies specific to those centers. Or suppose certain doctors were more likely to take melanoma images, and certain doctors were more likely to take non-melanoma images, and…
There _could_ be.
But when the source dataset was carefully gathered for a competition and the academics are saying "Broad and international participation in image contribution ensures that the dataset contains a representative clinically relevant sample", talking about multiple different equipment and labs, things look pretty promising.
A lot of ML systems are built with much less rigourous datasets and do a good job when you put them in production.
10k such images were gathered from this process.
The authors then randomly select 100 images from it to use as a test dataset. 100 is a small number. But that smallness is not relevant to the _selection_ issues here. Its only relevant to whether the measurements of performance are statistically significant (i.e. that we didn't end up with a sample that by chance is particularly favorable to the ML approach.) 100 data points is enough that that's unlikely (though one should check.)
Additionally the authors talk about using a much larger validation set, and performing multiple runs and checking the validation accuracy is similar. Unless they deliberately left out the damning fact that their accuracy was a lot _higher_ on their small test set than on their validation set, then its even more unlikely that the test set is a sample that is particularly favorable to the ML approach.
You could argue it happens to be particularly _unfavorable_ to the humans, but that seems a stretch. Perhaps they should have created multiple test sets, and a couple of different batches of human raters etc. But honestly, their setup seems pretty good to me.
> (a different comment is the one calling the researchers idiots). That's fair.
I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…
I do research in computer vision and this paper is so bad it's beyond words. * They give the network is huge advantage: they teach it that it should say "no" 80% of the time. The training data is unbalanced (80% no vs 20% yes) as is the test data. Of course it does well! I don't care what they do at training time, but the test data should be balanced or they should correct for this in the analysis. * They measure the…
Why would you ever balance your test data? If 80/20 is the actual population distribution, the sample that forms your test set should conform to that. Balance all you want in train/validation sets, but never the test set. Not balancing and using ROC is a terrible combo, but the metric is the problem, not the lack of artificial balance.