41–50 of 64 posts
Our company is very thankful for the models developed for a competition we held through kaggle. The model is in our current release product, and is the current known state of the art for it's task in accuracy / performance.
Earlier quoted context omitted.
One of the authors of above paper here. There is a whole line of work that supports the notion that all progress we are making via this leaderboard (or competition) mechanism is in fact real progress -- and that we are not simply overfitting to the test set. This thread by Moritz Hardt does a good job laying a few reasons why this may the case: https://twitter.com/mrtz/status/1134158716251516928
I'd like to signal boost this, it's a very important line of research. Congrats on a groundbreaking paper! The first time I have seen it, I was completely shocked by that perfect linear fit on "Do ImageNet Classifiers Generalize to ImageNet?" Figure 1. To elaborate for the other commenters: vaishaalshankar's team has created a new ImageNet evaluation dataset from scratch, and they observed that the leaderboard positi…
That paper shows that imagenet classifiers retain their ranking on data drawn from the same distribution as the training set. That isn't even close to the same thing as generalising to real world data. Medical AI shows us over and over again that truly out of distribution unseen data (external validation) is a completely different challenge to simply drawing multiple test sets from your home clinic.
Again, I don't actually think imagenet was as problematic as other competitions, but there is better evidence for that (not the least of which is that for the first half of imagenet's life, the differences in models were large, the number of tests was cumulatively fairly small, and the test set was huge: ie what I wrote supports imagenet as fairly reliable).
Our company is very thankful for the models developed for a competition we held through kaggle. The model is in our current release product, and is the current known state of the art for it's task in accuracy / performance.
When did someone last really pick it apart in its operational environment?
Earlier quoted context omitted.
Yes, it's a very thought-provoking article. I'm sure there are many competitions on Kaggle that were won due to the testing/training splits or other incidental choices rather than better machine learning. But I don't think it's fair to complain about $30,000 prizes being awarded to first rather than second place in a specific competition without doing at least a little checking of whether that was actually the case.…
All I can really say is that my usual readers understand that I am pro-ML, in fact I'm probably more hung go about the potential of deep learning than many of my compatriots. I've fallen victim of getting a Twitter bump, and assuming that people know I'm not anti-ML. The blog post is meant to be educational, not argumentative. Since it has got wider exposure I'll do a follow up to clarify my position on imagenet.
Two reasons : 1 - it's harder to do this vs. optimise the behooozas out of a dataset and throw the best one over the wall (and this is often done in good heart complete with a whole gamut of "standard practice" which are in-fact information leak from test to train like checking what features are informative on the test set before doing training) 2... folks don't know better, and best practice is sparsely documented or taught. This is because there are almost no practitioners turned teachers in comp sci. I'm not running down the great people who do great work pushing the field, they are my betters, but the next generation are being mislead into thinking that the skills they are picking up in their ML classes are going to keep them gainfully employed in the long term.
I guess the calculator he used is https://select-statistics.co.uk/calculators/sample-size-calc... . The calculator includes this caveat: > "If one or both of the sample proportions are close to 0 or 1 then this approximation is not valid and you need to consider an alternative sample size calculation method." 0.86 is fairly close to 1. And they're not proportions but rather averages of Dice coefficients. The statisti…
A common (industry) scenario is that you have a classifier that is 99.xx or 5x.xx accurate (it makes very few mistakes but they are costly, or it's a bit better than a coin flip, but we'll take that as it's what pays the mortgage), and we need to be absolutely positively certain that the one that we are fielding really works and is the best one we have (or homeless-a-go-go)
With the calculator
99.79 vs 99.65 @99% & 99% power-> 68457 examples
50.79 vs 50.65 @99 &99 -> 6129161 examples
which is why fancy models with marginal demonstrated improvement are often kept in the draw - much to the frustration of ML folks who are sure that it works and have proved that it will make the firm $10M a week.
This is a bit like saying "F1 doesn't produce useful cars", isn't it? The point of a competition is to meet specific parameters as well as possible and push the boundaries of what can be done. It's not meant to create a "daily driver". I realize he argued against this with the coin flip test, but that is why you'd ideally want to have many of these competitions over time. If you start to see the same names popping up…
My understanding is that the context is "usable models in clinical setting". Am I reading it wrong?
Earlier quoted context omitted.
Yes, it's a very thought-provoking article. I'm sure there are many competitions on Kaggle that were won due to the testing/training splits or other incidental choices rather than better machine learning. But I don't think it's fair to complain about $30,000 prizes being awarded to first rather than second place in a specific competition without doing at least a little checking of whether that was actually the case.…
All I can really say is that my usual readers understand that I am pro-ML, in fact I'm probably more hung go about the potential of deep learning than many of my compatriots. I've fallen victim of getting a Twitter bump, and assuming that people know I'm not anti-ML. The blog post is meant to be educational, not argumentative. Since it has got wider exposure I'll do a follow up to clarify my position on imagenet.
Earlier quoted context omitted.
I'd like to signal boost this, it's a very important line of research. Congrats on a groundbreaking paper! The first time I have seen it, I was completely shocked by that perfect linear fit on "Do ImageNet Classifiers Generalize to ImageNet?" Figure 1. To elaborate for the other commenters: vaishaalshankar's team has created a new ImageNet evaluation dataset from scratch, and they observed that the leaderboard positi…
I don't actually think imagenet is anywhere near as susceptible to crowd based overfitting as most kaggle competitions, but I don't actually think that paper falsifies the claim that it is. That paper shows that imagenet classifiers retain their ranking on data drawn from the same distribution as the training set. That isn't even close to the same thing as generalising to real world data. Medical AI shows us over and…
Not the same distribution, it's new data collected and processed according to the same recipe. A quite different distribution, demonstrated by the fact that the accuracy numbers drop sharply. That's why it's so surprising that the rankings do not change that much. (Okay, in principle, a possible explanation is that it is the exact same distribution, with a fixed percentage of mislabeled or impossibly hard-to-label datapoints added. Appendix B2 of the paper deals with this possibility.)
In any case, I fully agree that this kind of generalization is still much easier than generalizing to real world data.
> Again, I don't actually think imagenet was as problematic as other competitions, but there is better evidence for that (not the least of which is that for the first half of imagenet's life, the differences in models were large, the number of tests was cumulatively fairly small, and the test set was huge: ie what I wrote supports imagenet as fairly reliable).
CIFAR-10 is basically the opposite of your list of requirements. Train set small, test set small, test set public, small number of labels, grid searched to death. And yet, look at the CIFAR-10 graph from that paper. The exact same pattern as ImageNet.
I'm always for examining replicability, but the current paradigm to me seems misguided and this articulates some of the reasons very well.