51–60 of 64 posts
The headline, apart from being clickbaity, is accepting the null hypothesis . Which is a STAT101 no no. The article makes several good points. But just because the testing isn’t sufficient to prove that the winner didn’t just get lucky, it doesn’t prove that the winner did just get lucky.
Earlier quoted context omitted.
I think the point is we are left with uncertainty. Your prior should be that we don't know which competitor is best, and after the competition we are still unsure.
Isn't that the same thing as "not producing useful models"? Like, sure, some of the models may work, but unless you know which ones you can't make use of them.
Winners have often developed really interesting feature engineering strategies for a domain, as well as very well organised tuning and stacking systems.
Maybe a lot of difference between 1st and 5th percentile is luck, but top end of Kaggle is a really valuable insight into building effective models, even if you might want to simplify implimentation a bit in the commercial world.
> Now imagine you aren’t flipping coins. Imagine you are all running a model on a competition test set. Instead of wondering if your coin is magic, you instead are hoping that your model is the best one, about to earn you $25,000. > Of course, you can’t submit more than one model. That would be cheating. One of the models could perform well, the equivalent of getting 8 heads with a fair coin, just by chance. > Good t…
What they're saying, as far as I understand it, is that the best k teams (where k << N) each win roughly 1/k of the time.
As someone who develops machine-learning models as job, I can say that there are snippets from Kaggle kernels that run in minutes, and outperform months of manual effort by a business analyst. Winners have often developed really interesting feature engineering strategies for a domain, as well as very well organised tuning and stacking systems. Maybe a lot of difference between 1st and 5th percentile is luck, but top…
Huh? How do you "overfit" on the test set if you don't have the test set? And also to call a good ML result a "coin toss" shows a profound lack of understanding of what goes into such "coin tosses", and why solving practical problems with ML is an entirely different ballgame than training a classifier on imagenet (which is in itself pretty hard if you want SOTA results).
By being lucky. And with a large enough number of solutions, at least one solution will almost surely be lucky.
The earlier example says that the difference between winners in an arbitrarily picked Kaggle competition was "0.0014". Sure, I agree, seems small. But this random diagram about image classification shows that the Google model "the improvement year on year slows (the effect size decreases)". But that's not even true. The effect is exponential!
2011->2012: 38.6% "Reliable Improvement" 2014->2015: 34.9% "Probably Overfitting"
Is this really the thoughts of someone well versed in statistics? I get the feeling they are just upset they lost at a competition and decided instead to rant about it (as you can see from the aggressive use of memes). This is not a well thought out argument against ML competitions, but you might be fooled into thinking it was because it contains just enough discussion of statistics that you might not notice it doesn't hold up.
Huh? How do you "overfit" on the test set if you don't have the test set? And also to call a good ML result a "coin toss" shows a profound lack of understanding of what goes into such "coin tosses", and why solving practical problems with ML is an entirely different ballgame than training a classifier on imagenet (which is in itself pretty hard if you want SOTA results).
> How do you "overfit" on the test set if you don't have the test set? By being lucky. And with a large enough number of solutions, at least one solution will almost surely be lucky.
Agree with a mild version of the author's statement: that for many competitions, the difference between top n spots is not statistically significant. However, the author's statement (as represented by this chart https://lukeoakdenrayner.files.wordpress.com/2019/09/ai-comp...) is far too strong.
The actual best model may not always win, but will typically be in the top 0.1%.
There are people on this thread who have poked holes in the author's sample size calculator (I'm not going to rehash that).
But an empirical observation: the same top ranked Kagglers consistently perform well in competition after competition.
You can see this by digging through the profiles of top ranked Kagglers (https://www.kaggle.com/rankings). Or by looking at competition leaderboards. For example the leaderboard screenshot the author shared in the post (https://lukeoakdenrayner.files.wordpress.com/2019/09/pneumo-...) shows 11 of the 13 top performers are Masters and Grandmasters, which puts them at the top ranked 1.5K members of our community of 3.4MM data scientists (orange and gold dots under the profile pictures indicate Master and Grandmaster rank).
I actually think the author's headline is often correct: there are many cases where machine learning competitions don't produce useful models. But for a completely different reason: Competitions sometimes have leakage.
Kaggle CEO here. Agree with a mild version of the author's statement: that for many competitions, the difference between top n spots is not statistically significant. However, the author's statement (as represented by this chart https://lukeoakdenrayner.files.wordpress.com/2019/09/ai-comp... ) is far too strong. The actual best model may not always win, but will typically be in the top 0.1%. There are people on this…
As a funny example: I remember we were once given a dataset to predict prostate cancer from ~300 variables. One of the variables was "had prostate cancer surgery". Turned out that was a very good predictor of prostate cancer ;). Thankfully that was an example where we caught the leakage. Unfortunately there are cases where we don't catch the leakage.
