Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Has anyone built a recommendation engine in-house?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Ask HN: Has anyone built a recommendation engine in-house?

#31

Post by superasn »

prades wrote: post #20

I've built a recommender system for my movie database app "Coollector Movie Database". It's based on Collaborative filtering and it took me 2 years to implement. I built it from scratch and it's unique in several ways (for example, you can view the reliability of each recommendation). The technical difficulty is to crunch fast enough a huge quantity of data. I've had to apply all the optimizations that I could think…

That is a pretty amazing site! Can't download anything right now which makes me think It would make it a lot easier if instead of a downloadable software it was being offered as a saas site.

Re: Ask HN: Has anyone built a recommendation engine in-house?

#32

Post by prades »

superasn wrote: post #31
prades wrote: post #20

I've built a recommender system for my movie database app "Coollector Movie Database". It's based on Collaborative filtering and it took me 2 years to implement. I built it from scratch and it's unique in several ways (for example, you can view the reliability of each recommendation). The technical difficulty is to crunch fast enough a huge quantity of data. I've had to apply all the optimizations that I could think…

That is a pretty amazing site! Can't download anything right now which makes me think It would make it a lot easier if instead of a downloadable software it was being offered as a saas site.

Thank you! It could not be free if it was a website because calculating the recommendations uses quite a lot of CPU. It's fine when run on each user's computer, but it would require expensive servers to make all the calculations for everyone, with a risk of congestion.

There's a website named criticker.com which gives great movie recommendations, but you'll see that they have problems handling all the calculations. You'll literally see the recommendations being slowly generated, and updates are a problem when you rate more movies.

Re: Ask HN: Has anyone built a recommendation engine in-house?

#33

Post by rwieruch »

I was in the same situation and learned about recommender systems by implementing a simple movie recommendation system in JavaScript. If you are interested, you can find the source code over here: https://github.com/javascript-machine-learning/movielens-rec...

Re: Ask HN: Has anyone built a recommendation engine in-house?

#34

Post by eggie5 »

lightFM is my goto for prototyping Matrix Factorization models. It efficiently handles large data w/ sparse data structures and is CPU accelerated including optimizations like Hogwild!. It also has the WARP loss BPR variant which I have not seen implemented anywhere else.

I can train on multi-GB datasets w/ only lightFM and multiple CPUs.

Another interesting package is called Implicit. This package, although not as complete as LightFM when it comes to algorithms or APIs, really shines when it comes down to optimizations. Including native Cuda kernels for BPR and ALS it also has an important speedup called the Conjugate Gradient Method which makes it faster than spark in some benchmarks.

But usually, now-a-days my work requires more customized hybrid models of which I usually start w/ a base BPR implementation I have in Keras.

Re: Ask HN: Has anyone built a recommendation engine in-house?

#35

Post by eggie5 »

chudi wrote: post #5

There are basically 3 types of recommender engines: Content Based: If you can represent your products as a vector, you can have a distance between each product, then you have a item-item recommendation. You can use all kinds of embedding to achieve this results, some techniques that we tried are word2vec embedding of user navigation, auto encoding of features using neural networks, dimensionality reduction with PCA,…

I don't understand your connection between lightfm and the youtube paper...

Re: Ask HN: Has anyone built a recommendation engine in-house?

#36

Post by eggie5 »

prades wrote: post #20

I've built a recommender system for my movie database app "Coollector Movie Database". It's based on Collaborative filtering and it took me 2 years to implement. I built it from scratch and it's unique in several ways (for example, you can view the reliability of each recommendation). The technical difficulty is to crunch fast enough a huge quantity of data. I've had to apply all the optimizations that I could think…

How would you generalize the method you are using?

Re: Ask HN: Has anyone built a recommendation engine in-house?

#38

Post by chudi »

eggie5 wrote: post #35
chudi wrote: post #5

There are basically 3 types of recommender engines: Content Based: If you can represent your products as a vector, you can have a distance between each product, then you have a item-item recommendation. You can use all kinds of embedding to achieve this results, some techniques that we tried are word2vec embedding of user navigation, auto encoding of features using neural networks, dimensionality reduction with PCA,…

I don't understand your connection between lightfm and the youtube paper...

they are hybrid in the sense that gather signals from not just features or user activity, yt paper uses embeddings from search and views, so its more of a mixed model than a pure one content based or a pure collaborative filtering

Re: Ask HN: Has anyone built a recommendation engine in-house?

#39

Post by [deleted] »

MaxLeiter wrote: post #8
sadikkapadia1 wrote: post #6

I wrote the recommendation system at Netflix (still in use after 5 years). Primary problem was company politics. Many groups were not happy that one person could write a system that was better in A/B test, had more uptime and cheaper to run. All of it (ML, production, monitoring), was custom code.

Almost all of your top-level comments mention you did this

[deleted]

Re: Ask HN: Has anyone built a recommendation engine in-house?

#40

Post by eggie5 »

chudi wrote: post #38
eggie5 wrote: post #35

Earlier quoted context omitted.

I don't understand your connection between lightfm and the youtube paper...

they are hybrid in the sense that gather signals from not just features or user activity, yt paper uses embeddings from search and views, so its more of a mixed model than a pure one content based or a pure collaborative filtering

ok, I see, you are making the connection on basis of hybrid characteristics.

Since you're familiar w/ the youtube paper, I've been wondering this question: How do they get vectors out of the softmax?

Post reply on HN