AI competitions don’t produce useful models
lukeoakdenrayner.wordpress.com
AI competitions don’t produce useful models
1–10 of 64 posts
Re: AI competitions don’t produce useful models
#2or are you a sore person who cannot compete in AI competitions??
Re: AI competitions don’t produce useful models
#3This is a bit like saying "F1 doesn't produce useful cars", isn't it?
The point of a competition is to meet specific parameters as well as possible and push the boundaries of what can be done. It's not meant to create a "daily driver".
I realize he argued against this with the coin flip test, but that is why you'd ideally want to have many of these competitions over time. If you start to see the same names popping up at the top regularly, you know there's some sort of significance to them. Teams would ultimately want to trend towards whatever wins competitions most consistently, so they'd want to rely on models they think are the most likely to perform in a real world test. They wouldn't want to simply rely on a coin flip.
And in large competitions, you have the chance of batching the top performers together and seeing what is common between them. Presumably there's a reason these models pan out over the rest in aggregate; are they worth pursuing a bit?
I think we agree at the end though, even within my analogy. A huge value of F1 racing is the publicity the teams give their sponsors. They might learn some information that can be pushed down to their consumer vehicles, but it's marginal compared to team winnings and the value of saying "See? Our engineers are the best".
Re: AI competitions don’t produce useful models
#4https://arxiv.org/abs/1902.10811 is an useful counterpoint to this article's comments on ImageNet overfitting (esp e.g. §3.3, "Few Changes in the Relative Order").
Re: AI competitions don’t produce useful models
#5There was a good discussion on Reddit on how the objective of Kaggle competitions from the competition creator's perspective isn't necessarily a useful model. https://www.reddit.com/r/MachineLearning/comments/d50lr3/d_w...
Re: AI competitions don’t produce useful models
#6It seems like the author is saying two things. 1) The hold out method isn't robust enough and 2) There may not be much difference between the top performing models. Which, okay, there may be a point there. But that doesn't really support the thesis that competitions don't produce useful models. It's more like they don't produce models that can be deployed directly for clinical use. There's no reason the techniques used by the competitors can't contribute to something that is clinically useful.
Of course a title like, "Here's why you can't directly deploy ML competition models into a production environment" doesn't grab as many clicks.
Re: AI competitions don’t produce useful models
#7My comment is really similar to what's already been said, this reads a bit like a "I don't really want to waste my time so neither should you" post. I mean nothing wrong with that but a bit negative and sour then I personally would like. Plus I think the entire spirit of the competition was a bit lost on this person.
Solid case in point, the u-net came to prominence from a medical kaggle competition. Was it a "useful" model? The author might not be wrong in saying the model wouldn't work as well in the wild but I would definitely say it was useful. The unet is still a very commonly used architecture
Re: AI competitions don’t produce useful models
#8Given that a handful of names occupy Gold places in kaggle competitions, I would not call it luck. Given how hard it is to stay at the top on private set, I would not call it luck.
The winning models are hardly ever used in production, but the set of skills needed to get a gold is.
Re: AI competitions don’t produce useful models
#9The headline, apart from being clickbaity, is accepting the null hypothesis. Which is a STAT101 no no.
The article makes several good points. But just because the testing isn’t sufficient to prove that the winner didn’t just get lucky, it doesn’t prove that the winner did just get lucky.
Re: AI competitions don’t produce useful models
#10Sigh this beautifully captures the tragedy of biostats and epidemiology which is held hostage by lack of systems thinking. Unlike say a clinical trials, ML competitions are not limited to data available at start of the competition in fact by having a fair measurement of performance there is strong incentive to label and share more data and run checkpointed models. Further since the goal is not “publishing” with arbitrary requirements of P values / Power an empirical strategy is likely to provide much better long term accountability and better models. Sadly the cult of bio stats is so deeply vested in publishing rather than thinking end to end (designing a system as whole) any other models is quickly rejected by the community.