21–30 of 64 posts
Earlier quoted context omitted.
I think the point is we are left with uncertainty. Your prior should be that we don't know which competitor is best, and after the competition we are still unsure.
Isn't that the same thing as "not producing useful models"? Like, sure, some of the models may work, but unless you know which ones you can't make use of them.
> "If one or both of the sample proportions are close to 0 or 1 then this approximation is not valid and you need to consider an alternative sample size calculation method."
0.86 is fairly close to 1. And they're not proportions but rather averages of Dice coefficients. The statistics used here seem very suspicious. I'd like to see a Monte Carlo simulation or at least the assumptions / derivation of the formula used.
I guess the calculator he used is https://select-statistics.co.uk/calculators/sample-size-calc... . The calculator includes this caveat: > "If one or both of the sample proportions are close to 0 or 1 then this approximation is not valid and you need to consider an alternative sample size calculation method." 0.86 is fairly close to 1. And they're not proportions but rather averages of Dice coefficients. The statisti…
No statistician would do what I did for a formal publication, but I think what I did gets the point across.
I guess the calculator he used is https://select-statistics.co.uk/calculators/sample-size-calc... . The calculator includes this caveat: > "If one or both of the sample proportions are close to 0 or 1 then this approximation is not valid and you need to consider an alternative sample size calculation method." 0.86 is fairly close to 1. And they're not proportions but rather averages of Dice coefficients. The statisti…
The post also doesn't draw on any actual reported experiences of either competitors or hosts of competitions. Generally in post competition wrap-ups there's a lot of information sharing and very significant development of understanding that comes about.
To take the imagenet example that's claimed in the post as being a likely example of over-fitting: each year's imagenet results led to very significant advances in our understanding of image classification, which are today very widely used in industry and research.
> Now imagine you aren’t flipping coins. Imagine you are all running a model on a competition test set. Instead of wondering if your coin is magic, you instead are hoping that your model is the best one, about to earn you $25,000. > Of course, you can’t submit more than one model. That would be cheating. One of the models could perform well, the equivalent of getting 8 heads with a fair coin, just by chance. > Good t…
The competition and test set will still have hidden biases due to the ontology you use.
Sufficient optimization pressure always eventually overcomes your bias control metrics, in the context of actual utility rather than other metrics.
It's well known in cryptography and security that all abstractions are leaky.