BelKor has allegedly won Netflix Prize despite being second on the leaderboard
1–10 of 20 posts
Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard
#2an .0001 difference would seem to me to be 'too close to call' and a reason to reopen and let both teams work for another 30 days to get to a larger than .1 difference.
Just like in ping-pong, extend the game if it is that close until there is a clear winner.
With a winner takes all contest like this there has to be a clear winner.
Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard
#3an .0001 difference would seem to me to be 'too close to call' and a reason to reopen and let both teams work for another 30 days to get to a larger than .1 difference. Just like in ping-pong, extend the game if it is that close until there is a clear winner. With a winner takes all contest like this there has to be a clear winner.
Yes, the difference is negligible, but you obviously can't change rules of the game when it has already finished.
Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard
#4an .0001 difference would seem to me to be 'too close to call' and a reason to reopen and let both teams work for another 30 days to get to a larger than .1 difference. Just like in ping-pong, extend the game if it is that close until there is a clear winner. With a winner takes all contest like this there has to be a clear winner.
Winner takes all contests seem to be zero sum games to me, in a certain sense. The pie is a fixed size, and one guy gets all of it. I think I'd rather sink my time into business or open source stuff.
Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard
#5an .0001 difference would seem to me to be 'too close to call' and a reason to reopen and let both teams work for another 30 days to get to a larger than .1 difference. Just like in ping-pong, extend the game if it is that close until there is a clear winner. With a winner takes all contest like this there has to be a clear winner.
Yes, the difference is negligible, but you obviously can't change rules of the game when it has already finished.
And to be clear: The winner of the contest is not determined by the number that you see listed on the leaderboard. The number on the leaderboard [1] is on the Quiz dataset, the winner is determined by running against the Test dataset (which is kept private by Netflix). As stated by Yehuda on team BellKor: "our team is top contender for winning the Grand Prize, as we have a better Test score than The Ensemble."
This turnaround does not surprise me. BellKor's Pragmatic Chaos took their sweet time getting to 10%+ - and in doing so they were very sure not to overfit [2] the data (making their solution a much more generic and viable solution to the dataset). It's my guess that The Ensemble rushed quickly to 10%+ and overfit their data like mad (which yielded high numbers on the public dataset, but evidently does not translate to a generic solution).
I'm looking forward to seeing the final papers published by BellKor, et. al. - they're going to be a fascinating read, regardless.
1: http://www.netflixprize.com/leaderboard 2: http://en.wikipedia.org/wiki/Overfitting
Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard
#6Earlier quoted context omitted.
Yes, the difference is negligible, but you obviously can't change rules of the game when it has already finished.
And to be clear: The winner of the contest is not determined by the number that you see listed on the leaderboard. The number on the leaderboard [1] is on the Quiz dataset, the winner is determined by running against the Test dataset (which is kept private by Netflix). As stated by Yehuda on team BellKor: "our team is top contender for winning the Grand Prize, as we have a better Test score than The Ensemble." This t…
A lot of people have been tripped up by this, and it's a VERY important distinction. I hope Netflix posts more about the final (Test) results from each when they officially announce the winner.
Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard
#7A contest like this one (with such a large prize at stake) should have seen this coming and instituted a different rule for their deadline. Something like "24 hours after the last submission." Therefore you wouldn't have someone sniping the win like it seems has happened.
Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard
#8an .0001 difference would seem to me to be 'too close to call' and a reason to reopen and let both teams work for another 30 days to get to a larger than .1 difference. Just like in ping-pong, extend the game if it is that close until there is a clear winner. With a winner takes all contest like this there has to be a clear winner.
Winner takes all contests seem to be zero sum games to me, in a certain sense. The pie is a fixed size, and one guy gets all of it. I think I'd rather sink my time into business or open source stuff.
Actually, the top teams have already started negotiating with other companies for the technology that they came up with. Knowledge is never zero-sum.
Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard
#9an .0001 difference would seem to me to be 'too close to call' and a reason to reopen and let both teams work for another 30 days to get to a larger than .1 difference. Just like in ping-pong, extend the game if it is that close until there is a clear winner. With a winner takes all contest like this there has to be a clear winner.
Winner takes all contests seem to be zero sum games to me, in a certain sense. The pie is a fixed size, and one guy gets all of it. I think I'd rather sink my time into business or open source stuff.
Sure, if you're only weighing the cash prize. Contests like this often involve competitors spending more than the prize, certainly in total among all entrants, but sometimes even by the single winner.
But it's clear there are many other motivations at play: the excitement of the competition, the chance to learn based on a real-world problem and dataset, the fame available even to those who reach goals other than final victory. And many of the results can be published as papers or open-source software, as well.
Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard
#10Earlier quoted context omitted.
Winner takes all contests seem to be zero sum games to me, in a certain sense. The pie is a fixed size, and one guy gets all of it. I think I'd rather sink my time into business or open source stuff.
Actually, the top teams have already started negotiating with other companies for the technology that they came up with. Knowledge is never zero-sum.
I wrote contest, not knowledge. The knowledge is external to the contest. Some contests, like this one, have lots of external benefits to the top guys, but "winner takes all contest" in the abstract sense seems to me to be a zero sum game, so I stand by what I said: it's generally better to sink your time into something where the "external stuff" is actually the main benefit.