Ask HN: Has anyone built a recommendation engine in-house? • Hacker News Board

Re: Ask HN: Has anyone built a recommendation engine in-house?

#21

Post by prades » Mon, Aug 13, 2018, 10:18 PM UTC

sadikkapadia1 wrote: post #18

sebst wrote: post #15

Earlier quoted context omitted.

> Any idea why Recommendation Engine as a Service has not picked up? These kind of services aren't so much exposed to the public and likely don't start at <100 bucks a month, which could be why those services are not that visible. However, there are some e-commerce services going in that direction, such like AgilOne...

It is hard to sell technology to companies when they have their own teams (often using free libraries). Embedded teams are always experts and will often discredit better technology. Often the only method of testing is A/B. These can easily be manipulated. For instance at Netflix (ignoring more blatant practices), P-hacking (run thousands of simulations and report ones that worked), and HARKing (come up with a hypothe…

> That is part of the reason the recommender has been degrading over the years.

Netflix is clearly promoting its original shows. Do you think your system is still in use now that they've moved to thumbs rating and percent match score?

Re: Ask HN: Has anyone built a recommendation engine in-house?

#22

Post by sadikkapadia1 » Mon, Aug 13, 2018, 10:35 PM UTC

prades wrote: post #21

sadikkapadia1 wrote: post #18

Earlier quoted context omitted.

It is hard to sell technology to companies when they have their own teams (often using free libraries). Embedded teams are always experts and will often discredit better technology. Often the only method of testing is A/B. These can easily be manipulated. For instance at Netflix (ignoring more blatant practices), P-hacking (run thousands of simulations and report ones that worked), and HARKing (come up with a hypothe…

> That is part of the reason the recommender has been degrading over the years. Netflix is clearly promoting its original shows. Do you think your system is still in use now that they've moved to thumbs rating and percent match score?

That is what I have been told. It is however clearly messed up.

Re: Ask HN: Has anyone built a recommendation engine in-house?

#23

Post by prades » Mon, Aug 13, 2018, 10:45 PM UTC

sadikkapadia1 wrote: post #22

prades wrote: post #21

Earlier quoted context omitted.

> That is part of the reason the recommender has been degrading over the years. Netflix is clearly promoting its original shows. Do you think your system is still in use now that they've moved to thumbs rating and percent match score?

That is what I have been told. It is however clearly messed up.

I feel for you. It must be a huge disappointment to see what they've done to your work. Netflix recommendation system used to be very good a few years ago.

Re: Ask HN: Has anyone built a recommendation engine in-house?

#24

Post by amirouche » Mon, Aug 13, 2018, 10:45 PM UTC

sadikkapadia1 wrote: post #6

I wrote the recommendation system at Netflix (still in use after 5 years). Primary problem was company politics. Many groups were not happy that one person could write a system that was better in A/B test, had more uptime and cheaper to run. All of it (ML, production, monitoring), was custom code.

Same situation with a smaller project. I still struggle to pull off it for the problem I face today. How did you succeed?

Re: Ask HN: Has anyone built a recommendation engine in-house?

#25

Post by fratlas » Mon, Aug 13, 2018, 10:50 PM UTC

sadikkapadia1 wrote: post #6

I wrote the recommendation system at Netflix (still in use after 5 years). Primary problem was company politics. Many groups were not happy that one person could write a system that was better in A/B test, had more uptime and cheaper to run. All of it (ML, production, monitoring), was custom code.

What was the tech stack? Did you use a graph db? I've built my own in neo4j for about 500M data points, but would love to know what you used.

Re: Ask HN: Has anyone built a recommendation engine in-house?

#26

Post by ecesena » Mon, Aug 13, 2018, 10:50 PM UTC

I built the one at Theneeds.com and, if you're interested, this is the one at Pinterest [1].

At Theneeds we were recommending news = fresh content based on user's interest and other features. Because the content is fresh, you can't easily have enough data for a proper collaborative system.

Our algo was essentially the Reddit algo, where a piece of content gets a rank based on time and log of score. Score in Reddit is the upvotes - downvotes. At Theneeds we had a more complex score including social signals (likes on fb / RT on tw) so we could compute a meaningful score also without a big community of users. The other difference wrt Reddit was having different scores and different paces (multipliers) based on categories of content, so for example news in tech and politics from newspapers were updating faster than news on travel from magazines. And by normalizing the ranks, you can merge multiple categories in one -- a feature that I think Reddit also added.

As for the code/stack, custom written in python. We were using Redis to cache user timelines using sorted sets (including the guest users, i.e. the default top news for each category). In Redis, you can merge sorted sets, and we used it as an efficient way to create the new timeline when a new user was signing up.

[1] https://medium.com/@Pinterest_Engineering/introducing-pixie-...

Edit: added more details about tech.

Re: Ask HN: Has anyone built a recommendation engine in-house?

#27

Post by prades » Mon, Aug 13, 2018, 11:29 PM UTC

sadikkapadia1 wrote: post #22

prades wrote: post #21

Earlier quoted context omitted.

> That is part of the reason the recommender has been degrading over the years. Netflix is clearly promoting its original shows. Do you think your system is still in use now that they've moved to thumbs rating and percent match score?

That is what I have been told. It is however clearly messed up.

https://news.ycombinator.com/item?id=15607383

Amazing story! I don't know if it's true or if you're delusional, but I'm inclined to believe you as it would explain why Netflix recommendation system went downhill instead of improving.

Re: Ask HN: Has anyone built a recommendation engine in-house?

#28

Post by itronitron » Tue, Aug 14, 2018, 1:51 AM UTC

Yes, I have implemented a few content-based recommendation engines (referring to Chudi's taxonomy). The biggest existential threat is dealing with colleagues that want to question your work for not using f^xyz method that they have heard about. Having a straightforward evaluation framework in place to evaluate your results will go a long way towards ensuring the adoption and longevity of what you create.

I grow my own analysis code but use search APIs for storage and access (Lucene or Algolia)

Re: Ask HN: Has anyone built a recommendation engine in-house?

#29

Post by bbulkow » Tue, Aug 14, 2018, 5:39 AM UTC

theknight wrote: post #13

sadikkapadia1 wrote: post #6

I wrote the recommendation system at Netflix (still in use after 5 years). Primary problem was company politics. Many groups were not happy that one person could write a system that was better in A/B test, had more uptime and cheaper to run. All of it (ML, production, monitoring), was custom code.

@sadikkapadia - Any idea why Recommendation Engine as a Service has not picked up? I realize that building a use-case specific recommendation engine is unique. However, I am wondering is there a recommendation engine as a service, which is similar to algolia available/possible. I see only 2 players - yusp and recombee. I'd appreciate any thought you have on this.

I worked for a recommendation engine as a service company called aggregate knowledge in 2007 when they got their second round of funding, 25m from kleiner. It was a remarkably lousy business, and they didn't do well.

There was really nothing wrong with the concept. A little JavaScript on the page, a bunch of back end magic, the ability to use a larger data pool because you are collecting from multiple sites.... But people wouldn't pay, and the engagement of recommendations was never that good.

The secret we learned was, after a bunch of math and research, was that 'best in this category and 3 closest adjacent categories works so well in retail that a naive algo did very well.

Better than the fancy math, which once associated the Koran with the sports illustrated swim suit edition ( and vice versa ).

There is a massive data sufficiency problem, and no company with the real data would go into this low end business. Small companies can't get enough data to be relevent.

The world is 10 years later, ml is better understood, so it might all be different now.... Pm me if you are interested in further detail

Re: Ask HN: Has anyone built a recommendation engine in-house?

#30

Post by seektable » Tue, Aug 14, 2018, 6:36 AM UTC

Have you checked Apache Mahout recommendation framework ( https://mahout.apache.org/docs/latest/algorithms/recommender... )? For 'small data' it can be used as Java library (algorithms for single-machine); if you prefer .NET C# port is also present: https://github.com/nreco/recommender . If you're new to collaborative filtering 'Apache Mahout in Action' book will help a much.