Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

This Psychologist Might Outsmart the Math Brains Competing for the Netflix Prize

wired.com

Post reply on HN
Print view View original

21–25 of 25 posts

Re: This Psychologist Might Outsmart the Math Brains Competing for the Netflix Prize

#21

Post by pchristensen »

andreyf wrote: post #20
andreyf wrote: post #18

Yet another psychological problem mathematicians too eagerly claimed as their own. Why not try to find patterns about the movies which appeal to a certain individual? Is it against the rules to use outside data (actors, directors, etc)?

From the Netflix prize FAQ: Why not provide other data about the movies, like genres, directors, or actors? We know others do. Again, Cinematch doesn’t currently use any of this data. Use it if you want. That seems like an easy target - if I've 5-starred every movie with Kevin Spacey, I probably will like anything with Kevin Spacey. Why not mine blogs and reviews, trying to find themes which are appealing to individu…

I think the limiting factor is the amount of computation required. When you take the number of variables (actors, directors, themes, box office gross, etc) x the number of values each of them have (dozens to thousands) x the number of users, that's a lot of crunching to do. Perhaps do some pre-crunching to find actors that have a correlation like that and then just check those. Apply the "Kevin Spacey index", if you will.

Re: This Psychologist Might Outsmart the Math Brains Competing for the Netflix Prize

#22

Post by cainus »

dood wrote: post #3

This article needed a lot more info about the psychologist's approach to make it interesting, it was 90% background and the barest hint at the end about what he is actually doing. For all we know he may have just made minor tweaks to some existing algorithm.

The article mentions tweaking the algorithm to take the timing of ratings into account. It gives the example that a person might rate two movies as 3/5, and a third they watch right after as a 4/5, because it was better than the previous two, but under normal(ized) circumstances that user would've given the 3rd movie a 3/5 too.

Soo.... it's an interesting notion, that there can be time-segment-based normalization of the data set. Team Bellkor/KorBell credit a big part of their gains to using the ordering of rankings, rather than the rankings themselves to test for similarity, so this guy's actually got a novel approach for normalizing the dataset. I really don't know how he can detect if he's dealing with the kind of person that is meticulous enough to make sure their new ratings take all their previous ratings into account, though, or if he's dealing with the kind of person that gives everything a rating from 4/5 to 5/5.. I wouldn't be surprised if he hits a wall because of this.

Really I think netflix would make a lot better strides towards their goals by improving their data collection technique. They could probably make great enhancements towards normalization just by saying "you gave this previous movie 3/5 stars. How do you rate this latest movie?" That way the data would be much more normalized. There are lots of possibilities for this sort of enhancement, so I'm not sure why they're only letting competitors look at the already collected dataset.

Re: This Psychologist Might Outsmart the Math Brains Competing for the Netflix Prize

#23

Post by cainus »

andreyf wrote: post #20
andreyf wrote: post #18

Yet another psychological problem mathematicians too eagerly claimed as their own. Why not try to find patterns about the movies which appeal to a certain individual? Is it against the rules to use outside data (actors, directors, etc)?

From the Netflix prize FAQ: Why not provide other data about the movies, like genres, directors, or actors? We know others do. Again, Cinematch doesn’t currently use any of this data. Use it if you want. That seems like an easy target - if I've 5-starred every movie with Kevin Spacey, I probably will like anything with Kevin Spacey. Why not mine blogs and reviews, trying to find themes which are appealing to individu…

It's probably just not that hot of a solution, believe it or not. The recommendation engine should be able to make much better associations between movies, without even being able to describe what those associations are. Looking at features like genre, director, actor etc, is like a spam filter looking for specific spammy words. As soon as the spammers start saying "p3n1s" instead or Eddy Murphy starts making family movies, it falls apart. Check out http://karmatics.com/docs/evolution-and-wisdom-of-crowds.htm... (scroll down to "3. Recommendation Systems" for the relevant part) for an explanation.

Re: This Psychologist Might Outsmart the Math Brains Competing for the Netflix Prize

#24

Post by toffer »

cainus wrote: post #23
andreyf wrote: post #20

Earlier quoted context omitted.

From the Netflix prize FAQ: Why not provide other data about the movies, like genres, directors, or actors? We know others do. Again, Cinematch doesn’t currently use any of this data. Use it if you want. That seems like an easy target - if I've 5-starred every movie with Kevin Spacey, I probably will like anything with Kevin Spacey. Why not mine blogs and reviews, trying to find themes which are appealing to individu…

It's probably just not that hot of a solution, believe it or not. The recommendation engine should be able to make much better associations between movies, without even being able to describe what those associations are. Looking at features like genre, director, actor etc, is like a spam filter looking for specific spammy words. As soon as the spammers start saying "p3n1s" instead or Eddy Murphy starts making family…

Great link. Thanks for posting that.
Post reply on HN