My Python Code for the Netflix Prize • Hacker News Board

Re: My Python Code for the Netflix Prize

#21

Post by bjano » Sat, Aug 25, 2012, 12:10 PM UTC

jdleesmiller wrote: post #11

I also worked on this at uni and had lots of fun -- those lessons certainly look familiar! We were trying to mine Wikipedia for more information on the movies. The code's here: http://code.google.com/p/wikipedia-netflix/wiki/WikipediaNet... It includes Wikipedia parsing stuff and a fairly fast C++ implementation of the very cool BellKor kNN algorithm.

I also tried mining Wikipedia, but I found it's much more reliable to use the categories of the movies instead of the links.

The links are very often unrelated: "unlike in the movie X", "who by then had already became famous playing X in movie Z", etc. Instead of using the entire article, I only used links from the very first sentence. (" X is an Italian 1984 drama starring Z, etc...") These links could be connected to categories.

The problem I didn't expect was that the "hierarchy" of the categories is a mess. Seems easy in concept: "comedy-drama films" is a subcategory of both "comedy films" and "drama films", so you can use all the parent categories. In practice you get to completely unrelated categories in a few steps. The biggest challenge was cleaning this mess up algorithmically. Once that was done, I had a very nice taxonomy of the movies.

The result didn't come close anywhere to the top contestants, but it was a very interesting learning experience.

Re: My Python Code for the Netflix Prize

#22

Post by JacobiX » Sat, Aug 25, 2012, 2:09 PM UTC

Thanks for posting this. Your code combines two successful approachs : a latent factor model (SVD) and a neighborhood model :)

Here's my implementation of a recommender algorithm in C if someone is interested : https://github.com/GHamrouni/Recommender

Re: My Python Code for the Netflix Prize

#23

Post by andreasvc » Sat, Aug 25, 2012, 3:08 PM UTC

Nitpick: binary blobs like .pyc and .so don't belong in a code repository. Instead you would put a makefile or setup.py to compile the .pyx files.

Re: My Python Code for the Netflix Prize

#24

Post by rabidsnail » Sat, Aug 25, 2012, 3:51 PM UTC

arekp wrote: post #18

I wrote a 195-page monograph on the Netflix Prize, for people interested in that sort of stuff: http://arek-paterek.com/book

Comic Sans?

Re: My Python Code for the Netflix Prize

#25

Post by arekp » Sat, Aug 25, 2012, 4:06 PM UTC

rabidsnail wrote: post #24

arekp wrote: post #18

I wrote a 195-page monograph on the Netflix Prize, for people interested in that sort of stuff: http://arek-paterek.com/book

Comic Sans?

right on :)

the beauty of self-publishing

Re: My Python Code for the Netflix Prize

#26

Post by richardlblair » Sat, Aug 25, 2012, 4:25 PM UTC

You indent by 8 characters.... I wanted to read your code but this will make my eyes bleed.

From pep 8: "Use 4 spaces per indentation level."

Re: My Python Code for the Netflix Prize

#27

Post by tlocke » Sat, Aug 25, 2012, 4:30 PM UTC

malkarouri wrote: post #17

tlocke wrote: post #13

I had a look though this and was confused by this line: ratings[ratings in https://github.com/alexbw/Netflix-Prize/blob/master/src/pred... Is it specific to NumPy? Or perhaps a Python trick I haven't seen before?

Specific to NumPy. Boolean indexing. http://www.scipy.org/Cookbook/Indexing#head-86055279f6592d36...

Got it, thanks for the info and pointers everyone.

Re: My Python Code for the Netflix Prize

#28

Post by surine » Sat, Aug 25, 2012, 4:50 PM UTC

Mine is at https://github.com/hbcdev/Netflix in C++, got to the top 500. It runs just inside my 8GB PC :)

Re: My Python Code for the Netflix Prize

#29

Post by beaumartinez » Sat, Aug 25, 2012, 8:26 PM UTC

richardlblair wrote: post #26

You indent by 8 characters.... I wanted to read your code but this will make my eyes bleed. From pep 8: "Use 4 spaces per indentation level."

Close but no cigar. He's indenting with tabs.

Re: My Python Code for the Netflix Prize

#30

Post by beagle3 » Sun, Aug 26, 2012, 8:04 PM UTC

textminer wrote: post #12

jdleesmiller wrote: post #11

I also worked on this at uni and had lots of fun -- those lessons certainly look familiar! We were trying to mine Wikipedia for more information on the movies. The code's here: http://code.google.com/p/wikipedia-netflix/wiki/WikipediaNet... It includes Wikipedia parsing stuff and a fairly fast C++ implementation of the very cool BellKor kNN algorithm.

Love each of the individual BellKor approaches ( http://www2.research.att.com/~volinsky/netflix/ProgressPrize... ) for finding recommendations in the space of movies or users-- an MDS embedding, a PCA whitening, an NMF factorization by alternating least squares. Each of those hunches seems like the true art in these problems. The blending 100 of them together is far less interesting to me, though. Yet that seems to b…

> Is there an art or science then to the subsequent blending?

You could regard this as an application of the "Smoothed expectation theorem", Saying E[X] = E[E[X|Y]]. That is, if you are trying to compute the expectation of something, you can make it depend on anything else, and compute the inner expectation with respect to that. Might seem trivial or useless, but it is wildly applicable and often significantly simplifies computations.

One of the practical implications is that if you're not sure about something (underlying model, specific parameters), just apply some prior distribution and compute the expectation over that -- it is essentially guaranteed* to provide a better result than trying to pick the correct setup.

Although I'm not sure what the interpretation here would be.

* - so long as the entropy of your prior is not more wrong than the entropy of your hyper-parameters. This is often the case.