With all of this technology applied, I am still disappointed by Netflix's recommendations – to the point of just giving up and doing something else.
Maybe you're just disappointed with Netflix's inventory, not their recommendations.
Real World Recommendation System
11–20 of 156 posts
Re: Real World Recommendation System
#12MANGA
Re: Real World Recommendation System
#13> a machine learning model is trained that takes in all these dozens of features and spits out a score (details on how such a model is trained to be covered in the next post).
This part was the one I was interested in. As most of the rest are obvious.
Re: Real World Recommendation System
#14With all of this technology applied, I am still disappointed by Netflix's recommendations – to the point of just giving up and doing something else.
I was actually pretty impressed the other day when searching for "shiloh" (which they didn't have) because it showed a bunch of "related" queries to other dog movies (they also didn't have). The available search results were a little lacking though.
Re: Real World Recommendation System
#15> a machine learning model is trained that takes in all these dozens of features and spits out a score (details on how such a model is trained to be covered in the next post). This part was the one I was interested in. As most of the rest are obvious.
Looks like FAANG in the title is just to get your attention. Details are missing.
Re: Real World Recommendation System
#16Isn't this obvious list-building promotion for a company (Fennel) that sells recommendation systems? "Fennel AI: Building and deploying real world recommendation systems in production Launched 18 hours ago" Caveat reader.
Nothing wrong with some content marketing. They provide value to people in return for getting exposure to their brand. Simple healthy quid pro quo
Re: Real World Recommendation System
#17> As a result, primary databases (e.g. MySQL, Mongo etc.) almost never work
I mean it does. As far as I'm aware Facebook's ad platform is mostly backed by hundreds of thousands of Mysql instances.
But more importantly this post really doesn't describe issues of scale.
Sure it has the stages of recommendation, that might or might not be correct, but it doesn't describe how all of those processes are scheduled, coordinated and communicate.
Stuff at scale is normally a result of tradeoffs, sure you can use a ML model to increase a retention metric by 5% but it costs an extra 350ms to generate and will quadruple the load on the backend during certain events.
What about the message passing, like is that one monolith making the recommendation (cuts down on latency kids!) or micro services, what happens if the message doesn't arrive, do you have a retry? what have you done to stop retry storms?
did you bound your queue properly?
none of this is covered, and my friends, that is 90% of the "architecture at scale" that matters.
Normally stuff at scale is "no clever shit" followed by "fine you can have that clever shit, just document it clearly, oh you've left" which descends into "god this is scary and exotic" finally leading to "lets spend half a billion making a new one with all the same mistakes."
Re: Real World Recommendation System
#18How FAANG actually builds their recommendation systems:
Millions of cores of compute, exabyte scale custom data stores. Good recommendations are expensive. If you try to build a similar system on AWS, you will spend a fortune.
Most recommender models just use co-occurrence as a seed, this can actually work pretty well on it’s own. If you want to get fancy then build up a vectorized form of the document with something like an an autoencoder, then use some approximate nearest neighbors to find documents close by. 95% of the compute and storage is just spent on calculating co-occurrence though.
Re: Real World Recommendation System
#19> a machine learning model is trained that takes in all these dozens of features and spits out a score (details on how such a model is trained to be covered in the next post). This part was the one I was interested in. As most of the rest are obvious.
(Disclaimer: I'm the author of the post)
Good feedback, noted. Will get the next post focused on training within the next couple of days.
Re: Real World Recommendation System
#20Is there any recommendation system people we actually happy with? They all seem to suck in my experience