Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI competitions don’t produce useful models

lukeoakdenrayner.wordpress.com

Post reply on HN
Print view View original

11–20 of 64 posts

Re: AI competitions don’t produce useful models

#11

Post by vaishaalshankar »

ajtulloch wrote: post #4

https://arxiv.org/abs/1902.10811 is an useful counterpoint to this article's comments on ImageNet overfitting (esp e.g. §3.3, "Few Changes in the Relative Order").

One of the authors of above paper here. There is a whole line of work that supports the notion that all progress we are making via this leaderboard (or competition) mechanism is in fact real progress -- and that we are not simply overfitting to the test set. This thread by Moritz Hardt does a good job laying a few reasons why this may the case: https://twitter.com/mrtz/status/1134158716251516928

Re: AI competitions don’t produce useful models

#12

Post by xenocyon »

Sileni wrote: post #3

This is a bit like saying "F1 doesn't produce useful cars", isn't it? The point of a competition is to meet specific parameters as well as possible and push the boundaries of what can be done. It's not meant to create a "daily driver". I realize he argued against this with the coin flip test, but that is why you'd ideally want to have many of these competitions over time. If you start to see the same names popping up…

The article isn't actually that trivial; it makes a very provocative and bold statement towards the end, essentially declaring that all stated progress in image classification in the last 5 years is questionable.

I don't know if I necessarily agree with that, but there is definitely a danger in evaluating models purely based on their predictive power when many models are being evaluated on a common dataset - which is exactly the mainstream practice in the world of deep learning research - and it is therefore wise to be wary, not just as individuals but as a community.

Re: AI competitions don’t produce useful models

#13

Post by scoopertrooper »

Sileni wrote: post #3

This is a bit like saying "F1 doesn't produce useful cars", isn't it? The point of a competition is to meet specific parameters as well as possible and push the boundaries of what can be done. It's not meant to create a "daily driver". I realize he argued against this with the coin flip test, but that is why you'd ideally want to have many of these competitions over time. If you start to see the same names popping up…

> Teams would ultimately want to trend towards whatever wins competitions most consistently

Could this just surface the teams that just submitted the most models? Maybe some sort of wins per a submission score could help with this?

Re: AI competitions don’t produce useful models

#14

Post by lukeor »

xenocyon wrote: post #12
Sileni wrote: post #3

This is a bit like saying "F1 doesn't produce useful cars", isn't it? The point of a competition is to meet specific parameters as well as possible and push the boundaries of what can be done. It's not meant to create a "daily driver". I realize he argued against this with the coin flip test, but that is why you'd ideally want to have many of these competitions over time. If you start to see the same names popping up…

The article isn't actually that trivial; it makes a very provocative and bold statement towards the end, essentially declaring that all stated progress in image classification in the last 5 years is questionable. I don't know if I necessarily agree with that, but there is definitely a danger in evaluating models purely based on their predictive power when many models are being evaluated on a common dataset - which is…

Hi, author here. I didn't actually mean to suggest that the last 5 years of performance improvement could be spurious. That clearly isn't true. I use resnets/densenets etc in my day to day work!

What the picture was trying to say is that, within a given year, the "winner" becomes less likely to be truly better than the second place team. Alexnet was clearly better than the alternative, even with Bonferroni adjusted significance thresholds. Less so by 2016/17.

I'm writing a follow up on imagenet in particular to address some of the nuance. It is very clearly not a representative example of ML competitions, but the same effects still apply to some extent (imo).

Re: AI competitions don’t produce useful models

#15

Post by mlnewbie »

mr_toad wrote: post #9

The headline, apart from being clickbaity, is accepting the null hypothesis . Which is a STAT101 no no. The article makes several good points. But just because the testing isn’t sufficient to prove that the winner didn’t just get lucky, it doesn’t prove that the winner did just get lucky.

Yeah, we can't disprove anything, yadda yadda.

If almost every competition on Kaggle has a winner that is not significantly better than the bulk of the field, then that is proof. Chance correlations leading to you not rejecting the null can only take you so far.

Re: AI competitions don’t produce useful models

#16

Post by ipsa »

There are so many things just so plain wrong about this (I attempted to respond, then had to stop), that I feel this post is more of an attempt to instill the frustration felt when the author attempted to compete and promptly got run over by some SotA- hungry boost-junkies from countries where the p-test is not part of the curriculum in schools. I really don't know how to constructively salvage this... Talk about the role of luck in games?

Re: AI competitions don’t produce useful models

#17

Post by skybrian »

mlnewbie wrote: post #15
mr_toad wrote: post #9

The headline, apart from being clickbaity, is accepting the null hypothesis . Which is a STAT101 no no. The article makes several good points. But just because the testing isn’t sufficient to prove that the winner didn’t just get lucky, it doesn’t prove that the winner did just get lucky.

Yeah, we can't disprove anything, yadda yadda. If almost every competition on Kaggle has a winner that is not significantly better than the bulk of the field, then that is proof. Chance correlations leading to you not rejecting the null can only take you so far.

I think the point is we are left with uncertainty. Your prior should be that we don't know which competitor is best, and after the competition we are still unsure.

Re: AI competitions don’t produce useful models

#18

Post by landryraccoon »

> Now imagine you aren’t flipping coins. Imagine you are all running a model on a competition test set. Instead of wondering if your coin is magic, you instead are hoping that your model is the best one, about to earn you $25,000.

> Of course, you can’t submit more than one model. That would be cheating. One of the models could perform well, the equivalent of getting 8 heads with a fair coin, just by chance.

> Good thing there is a rule against it submitting multiple models, or any one of the other 99 participants and their 99 models could win, just by being lucky

I wonder what the author must think of Poker tournaments. Even assuming there is luck involved, unless all of the models are equally bad (which would be surprising) teams that produce better models should win much more than their fair share, where fair share is 1/N when N is the total number of submissions.

But lets say that the author is correct that ML competitions are mostly luck. That is a testable hypothesis - in particular, we would expect little to no correlation between the credentials of the competitors and their ranking in the competition. Is that actually the case? Do unknown individuals who just started doing Machine Learning win on their first Kaggle submission? If the author's hypothesis is correct, one would expect that to happen fairly often, and one would expect that even highly expert competitors should win approximately (only) their fair share.

Re: AI competitions don’t produce useful models

#19

Post by mlnewbie »

skybrian wrote: post #17
mlnewbie wrote: post #15

Earlier quoted context omitted.

Yeah, we can't disprove anything, yadda yadda. If almost every competition on Kaggle has a winner that is not significantly better than the bulk of the field, then that is proof. Chance correlations leading to you not rejecting the null can only take you so far.

I think the point is we are left with uncertainty. Your prior should be that we don't know which competitor is best, and after the competition we are still unsure.

Isn't that the same thing as "not producing useful models"? Like, sure, some of the models may work, but unless you know which ones you can't make use of them.
Post reply on HN