My Python Code for the Netflix Prize

#1

Post by alexbw » Fri, Aug 24, 2012, 11:40 PM UTC

I competed alone in the Netflix Prize in college under the team name "Hi!". I've never seen anybody release their code, and I'm getting back into machine learning now, and realized that some folks might want to take a gander at a competitive machine learning codeset.

It's implemented mostly in Python, with Cython for the real speed-sensitive parts (everything in file "svd.pyx" did the heavy lifting, and got me up the leaderboard).

I hope that some folks will find this useful.

My Python Code for the Netflix Prize
github.com

Re: My Python Code for the Netflix Prize

#2

Post by [deleted] » Sat, Aug 25, 2012, 12:58 AM UTC

[deleted]

Re: My Python Code for the Netflix Prize

#3

Post by tuananh » Sat, Aug 25, 2012, 1:18 AM UTC

great for ref. however i can't found the dataset anywhere on the internet :(

Re: My Python Code for the Netflix Prize

#4

Post by alexholehouse » Sat, Aug 25, 2012, 1:38 AM UTC

tuananh wrote: post #3

great for ref. however i can't found the dataset anywhere on the internet :(

Netflix had to pull the dataset after a researchers at U Texas were able to de-anonymize the dataset, shown in their paper Robust De-anonymization of Large Sparse Datasets.[http://www.cs.utexas.edu/~shmat/shmat_oak08netflix.pdf]

Re: My Python Code for the Netflix Prize

#5

Post by viraj_shah » Sat, Aug 25, 2012, 2:05 AM UTC

This was incredibly kind of you to post up. It is great to see it public domain as many can learn from it. The Cython code looks scary though - 18k lines! May I ask how long you spent on this?

Re: My Python Code for the Netflix Prize

#6

Post by alexbw » Sat, Aug 25, 2012, 2:12 AM UTC

@tuananh I've got the dataset stored away, but I don't know if I'm legally allowed to post it. Would love if someone could produce proof one way or the other.

@viraj_shah I spent about 6 months working on the project before I had to stop to concentrate on my schoolwork (I was a senior in collge at the time). I think it would have been impossible to do this for myself without Cython. If it were to happen today, I would probably be writing in PyCuda, or with Numba, and it would be much, much, MUCH more succinct.

Re: My Python Code for the Netflix Prize

#7

Post by viraj_shah » Sat, Aug 25, 2012, 2:15 AM UTC

alexbw wrote: post #6

@tuananh I've got the dataset stored away, but I don't know if I'm legally allowed to post it. Would love if someone could produce proof one way or the other. @viraj_shah I spent about 6 months working on the project before I had to stop to concentrate on my schoolwork (I was a senior in collge at the time). I think it would have been impossible to do this for myself without Cython. If it were to happen today, I woul…

@alexbw Thanks for the info, great work!

Re: My Python Code for the Netflix Prize

#8

Post by Nogwater » Sat, Aug 25, 2012, 2:35 AM UTC

Here's mine if anyone is interested. I wrote it in D and haven't looked at it in years. I'm sure it's not usable as-is, but it might be fun anyway.

https://github.com/nogwater/NetflixPrizeD

The algorithm is based on Simon Funk's blog post here: http://sifter.org/~simon/journal/20061211.html

For me, the best part was squeezing the data and indexes into memory. :)

Re: My Python Code for the Netflix Prize

#9

Post by skystorm » Sat, Aug 25, 2012, 2:47 AM UTC

Very nice. It might be helpful to (briefly) describe the actual techniques you tried in the readme file? At least that's the first thing I looked for...

Re: My Python Code for the Netflix Prize

#10

Post by raheemm » Sat, Aug 25, 2012, 5:01 AM UTC

Good readme file. Liked the lessons learned.