My Python Code for the Netflix Prize • Hacker News Board

Re: My Python Code for the Netflix Prize

#11

Post by jdleesmiller » Sat, Aug 25, 2012, 7:13 AM UTC

I also worked on this at uni and had lots of fun -- those lessons certainly look familiar! We were trying to mine Wikipedia for more information on the movies. The code's here:

http://code.google.com/p/wikipedia-netflix/wiki/WikipediaNet...

It includes Wikipedia parsing stuff and a fairly fast C++ implementation of the very cool BellKor kNN algorithm.

Re: My Python Code for the Netflix Prize

#12

Post by textminer » Sat, Aug 25, 2012, 8:07 AM UTC

jdleesmiller wrote: post #11

I also worked on this at uni and had lots of fun -- those lessons certainly look familiar! We were trying to mine Wikipedia for more information on the movies. The code's here: http://code.google.com/p/wikipedia-netflix/wiki/WikipediaNet... It includes Wikipedia parsing stuff and a fairly fast C++ implementation of the very cool BellKor kNN algorithm.

Love each of the individual BellKor approaches (http://www2.research.att.com/~volinsky/netflix/ProgressPrize...) for finding recommendations in the space of movies or users-- an MDS embedding, a PCA whitening, an NMF factorization by alternating least squares. Each of those hunches seems like the true art in these problems. The blending 100 of them together is far less interesting to me, though.

Yet that seems to be the sort of jockeying and tweaking these problems (seen now in Kaggle contests) seem to require. Is there an art or science then to the subsequent blending? Does one develop a better intuition for the problem at that point, or am I entirely missing the point of most ensemble methods (predictiveness over parsimonious understanding)?

Re: My Python Code for the Netflix Prize

#13

Post by tlocke » Sat, Aug 25, 2012, 8:19 AM UTC

I had a look though this and was confused by this line:

ratings[ratingsin

https://github.com/alexbw/Netflix-Prize/blob/master/src/pred...

Is it specific to NumPy? Or perhaps a Python trick I haven't seen before?

Re: My Python Code for the Netflix Prize

#14

Post by marklit » Sat, Aug 25, 2012, 8:26 AM UTC

Does anyone have any thoughts on the lack of conformity to PEP8? His code works and it's valid python but I feel that it's difficult to read.

Re: My Python Code for the Netflix Prize

#15

Post by jdleesmiller » Sat, Aug 25, 2012, 8:36 AM UTC

textminer wrote: post #12

jdleesmiller wrote: post #11

I also worked on this at uni and had lots of fun -- those lessons certainly look familiar! We were trying to mine Wikipedia for more information on the movies. The code's here: http://code.google.com/p/wikipedia-netflix/wiki/WikipediaNet... It includes Wikipedia parsing stuff and a fairly fast C++ implementation of the very cool BellKor kNN algorithm.

Love each of the individual BellKor approaches ( http://www2.research.att.com/~volinsky/netflix/ProgressPrize... ) for finding recommendations in the space of movies or users-- an MDS embedding, a PCA whitening, an NMF factorization by alternating least squares. Each of those hunches seems like the true art in these problems. The blending 100 of them together is far less interesting to me, though. Yet that seems to b…

I don't know of any deeper interpretation of blending. I guess one could look to the 'wisdom of crowds' for anecdotal evidence. There might be connections with consensus and voting systems, but those are mostly discrete AFAIK. It is at least a pragmatic way of exploiting the different biases of the members in the ensemble. In our case, the Netflix predictions we made based on Wikipedia data scored worse on their own than ratings-only predictions, but they attracted some weight in the blend and made the overall score (marginally) better.

Re: My Python Code for the Netflix Prize

#16

Post by jdleesmiller » Sat, Aug 25, 2012, 8:40 AM UTC

tlocke wrote: post #13

I had a look though this and was confused by this line: ratings[ratings in https://github.com/alexbw/Netflix-Prize/blob/master/src/pred... Is it specific to NumPy? Or perhaps a Python trick I haven't seen before?

I believe that's an example of 'logical indexing', which is a useful feature in many vector-oriented languages / packages:

http://blogs.mathworks.com/steve/2008/01/28/logical-indexing...

Re: My Python Code for the Netflix Prize

#17

Post by malkarouri » Sat, Aug 25, 2012, 9:47 AM UTC

tlocke wrote: post #13

I had a look though this and was confused by this line: ratings[ratings in https://github.com/alexbw/Netflix-Prize/blob/master/src/pred... Is it specific to NumPy? Or perhaps a Python trick I haven't seen before?

Specific to NumPy. Boolean indexing.

http://www.scipy.org/Cookbook/Indexing#head-86055279f6592d36...

Re: My Python Code for the Netflix Prize

#18

Post by arekp » Sat, Aug 25, 2012, 10:33 AM UTC

I wrote a 195-page monograph on the Netflix Prize, for people interested in that sort of stuff: http://arek-paterek.com/book

Re: My Python Code for the Netflix Prize

#19

Post by beambot » Sat, Aug 25, 2012, 10:46 AM UTC

malkarouri wrote: post #17

tlocke wrote: post #13

I had a look though this and was confused by this line: ratings[ratings in https://github.com/alexbw/Netflix-Prize/blob/master/src/pred... Is it specific to NumPy? Or perhaps a Python trick I haven't seen before?

Specific to NumPy. Boolean indexing. http://www.scipy.org/Cookbook/Indexing#head-86055279f6592d36...

Yep. The indexing and slicing are quite powerful. For those familiar with MATLAB, this is a fantastic resource: http://www.scipy.org/NumPy_for_Matlab_Users

Re: My Python Code for the Netflix Prize

#20

Post by vladekm » Sat, Aug 25, 2012, 11:28 AM UTC

PEP8