Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BelKor has allegedly won Netflix Prize despite being second on the leaderboard

netflixprize.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard

#11

Post by mquander »

jacquesm wrote: post #2

an .0001 difference would seem to me to be 'too close to call' and a reason to reopen and let both teams work for another 30 days to get to a larger than .1 difference. Just like in ping-pong, extend the game if it is that close until there is a clear winner. With a winner takes all contest like this there has to be a clear winner.

Given that further work on the dataset has continued to produce rapidly diminishing returns, one would assume that prolonging the contest would be likely to result in the teams being closer, not further apart.

Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard

#12

Post by halo »

davidw wrote: post #4
jacquesm wrote: post #2

an .0001 difference would seem to me to be 'too close to call' and a reason to reopen and let both teams work for another 30 days to get to a larger than .1 difference. Just like in ping-pong, extend the game if it is that close until there is a clear winner. With a winner takes all contest like this there has to be a clear winner.

Winner takes all contests seem to be zero sum games to me, in a certain sense. The pie is a fixed size, and one guy gets all of it. I think I'd rather sink my time into business or open source stuff.

Saying a winner-takes-all contest is "zero sum" in any sense is utterly nonsensical, as it's clearly not zero sum since one group wins $1,000,000.

Also, "one guy" doesn't necessarily get all of it, as team co-operation is implicitly encouraged as it increases the chances of winning. Additionally, the game is strictly skill-based, so the more skilled you are the more likely you are to win, which may encourage you to work in the contest. And that's excluding all the additional externalities involved.

Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard

#13

Post by davidw »

halo wrote: post #12
davidw wrote: post #4

Earlier quoted context omitted.

Winner takes all contests seem to be zero sum games to me, in a certain sense. The pie is a fixed size, and one guy gets all of it. I think I'd rather sink my time into business or open source stuff.

Saying a winner-takes-all contest is "zero sum" in any sense is utterly nonsensical, as it's clearly not zero sum since one group wins $1,000,000. Also, "one guy" doesn't necessarily get all of it, as team co-operation is implicitly encouraged as it increases the chances of winning. Additionally, the game is strictly skill-based, so the more skilled you are the more likely you are to win, which may encourage you to w…

> clearly not zero sum

Ok, so call it "fixed sum" or whatever else is technically correct - that's the point I'm trying to make. If one guy wins, another guy necessarily loses. That's not true in plenty of other things.

> the more skilled you are the more likely you are to win

Yes, but the second and third guys are quite skilled too, and win nothing (and yes, we are excluding the externalities since we're talking about the general case, not this specific one).

Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard

#14

Post by mhb »

davidw wrote: post #10
hvs wrote: post #8

Earlier quoted context omitted.

Actually, the top teams have already started negotiating with other companies for the technology that they came up with. Knowledge is never zero-sum.

I wrote contest, not knowledge. The knowledge is external to the contest. Some contests, like this one, have lots of external benefits to the top guys, but "winner takes all contest" in the abstract sense seems to me to be a zero sum game, so I stand by what I said: it's generally better to sink your time into something where the "external stuff" is actually the main benefit.

The term "zero sum" does not really make sense here. Netflix obviously thinks they're getting more than what they're spending on the contest and the winning team obviously thinks they're receiving more than what they've invested. So between those two parties, there is a net gain as is the case in most voluntary transactions.

At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either.

Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard

#15

Post by lliiffee »

jeresig wrote: post #5
vang3lis wrote: post #3

Earlier quoted context omitted.

Yes, the difference is negligible, but you obviously can't change rules of the game when it has already finished.

And to be clear: The winner of the contest is not determined by the number that you see listed on the leaderboard. The number on the leaderboard [1] is on the Quiz dataset, the winner is determined by running against the Test dataset (which is kept private by Netflix). As stated by Yehuda on team BellKor: "our team is top contender for winning the Grand Prize, as we have a better Test score than The Ensemble." This t…

I don't think that is correct. The numbers on the leaderboard are not the scores on the training set.

The netflix prize uses three datasets: the training set, the leaderboard set, and the test set. The training set is distributed to everyone. The test set is totally secret. Access to the leaderboard set is only by submitting results (once per 24 hours) and looking at the results. It is not at all trivial to "overfit" to this leaderboard set. (It could be done by, e.g. submitting results with slight tweaks to the algorithm parameters, but this would take a lot of time since you can only submit every 24 hours. Also, you would basically have to do it consciously.)

Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard

#16

Post by davidw »

mhb wrote: post #14
davidw wrote: post #10

Earlier quoted context omitted.

I wrote contest, not knowledge. The knowledge is external to the contest. Some contests, like this one, have lots of external benefits to the top guys, but "winner takes all contest" in the abstract sense seems to me to be a zero sum game, so I stand by what I said: it's generally better to sink your time into something where the "external stuff" is actually the main benefit.

The term "zero sum" does not really make sense here. Netflix obviously thinks they're getting more than what they're spending on the contest and the winning team obviously thinks they're receiving more than what they've invested. So between those two parties, there is a net gain as is the case in most voluntary transactions. At best, what you're saying is that other competitors have wasted their time, but I don't thi…

> Netflix ... here

For the third time: I am talking about the abstract case, not 'here', and I am talking about the point of view of the competitors, not the entity running the contest.

> At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either.

They haven't won anything, and unless there are externalities, they have wasted their time. How can they not have? They received nothing.

So, what I said is true: the contest is a "zero" or "fixed" sum game, and the other competitors gain only through externalities. You can downmod that all you like, but this more or less concurs with the definition of "zero sum game" in wikipedia.

Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard

#17

Post by Herring »

joshfinnie wrote: post #7

A contest like this one (with such a large prize at stake) should have seen this coming and instituted a different rule for their deadline. Something like "24 hours after the last submission." Therefore you wouldn't have someone sniping the win like it seems has happened.

But this isn't an auction. You can't just conjure up a better result once you know you've been surpassed. Belkor etc was probably working just as hard all month & the ensemble just got lucky.

Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard

#18

Post by mhb »

davidw wrote: post #16
mhb wrote: post #14

Earlier quoted context omitted.

The term "zero sum" does not really make sense here. Netflix obviously thinks they're getting more than what they're spending on the contest and the winning team obviously thinks they're receiving more than what they've invested. So between those two parties, there is a net gain as is the case in most voluntary transactions. At best, what you're saying is that other competitors have wasted their time, but I don't thi…

> Netflix ... here For the third time: I am talking about the abstract case, not 'here', and I am talking about the point of view of the competitors, not the entity running the contest. > At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either. They haven't won anything, and unless there are externalities, they have wasted their time. How can they not have? T…

The reason you are having to dogmatically repeat your idiosyncratic use of the term "zero sum" is that you are using it in such a narrow sense that it adds nothing to what you are saying. A typical example of a zero sum game is a futures contract which is contrasted with a non-zero sum game like stock investing.

If you exclude externalities like all parties in a transaction being happier with the outcome or placing present value on some possible future outcome, then there is no net positive sum game.

Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard

#19

Post by davidw »

mhb wrote: post #18
davidw wrote: post #16

Earlier quoted context omitted.

> Netflix ... here For the third time: I am talking about the abstract case, not 'here', and I am talking about the point of view of the competitors, not the entity running the contest. > At best, what you're saying is that other competitors have wasted their time, but I don't think that's true either. They haven't won anything, and unless there are externalities, they have wasted their time. How can they not have? T…

The reason you are having to dogmatically repeat your idiosyncratic use of the term "zero sum" is that you are using it in such a narrow sense that it adds nothing to what you are saying. A typical example of a zero sum game is a futures contract which is contrasted with a non-zero sum game like stock investing. If you exclude externalities like all parties in a transaction being happier with the outcome or placing p…

My view is that a lot of contest type things are not a good deal for the participants, and that it's a good idea to recognize that, going in. If there are external benefits, well, then it may be worth it. Otherwise, you're probably better off investing your time in something that's positive sum, such as, say, a startup. I would have thought that people here would get that.

Re: BelKor has allegedly won Netflix Prize despite being second on the leaderboard

#20

Post by jibiki »

halo wrote: post #12
davidw wrote: post #4

Earlier quoted context omitted.

Winner takes all contests seem to be zero sum games to me, in a certain sense. The pie is a fixed size, and one guy gets all of it. I think I'd rather sink my time into business or open source stuff.

Saying a winner-takes-all contest is "zero sum" in any sense is utterly nonsensical, as it's clearly not zero sum since one group wins $1,000,000. Also, "one guy" doesn't necessarily get all of it, as team co-operation is implicitly encouraged as it increases the chances of winning. Additionally, the game is strictly skill-based, so the more skilled you are the more likely you are to win, which may encourage you to w…

> Saying a winner-takes-all contest is "zero sum" in any sense is utterly nonsensical

I thought that zero sum just meant that the total score of all players is a constant, regardless of the strategies chosen?

Post reply on HN