Live data from Hacker News

Real World Recommendation System

blog.fennel.ai

121–130 of 156 posts

Re: Real World Recommendation System

#121

Earlier quoted context omitted.

> . As far as I'm aware Facebook's ad platform is mostly backed by hundreds of thousands of Mysql instances. Same for YouTube itself https://www.mysql.com/customers/view/?id=750 and they use Vitess for horizontal scaling: https://vitess.io/

YouTube has since migrated to Spanner, there’s a podcast episode with one of the Vitess creators that covers the politics of the switch

That sounds interesting — do you have a link?

Re: Real World Recommendation System

#122
post #113
post #94

Earlier quoted context omitted.

That was my thinking - anything of value is product-specific and behind closed doors. It's not my field, but something I see come up from time to time that seems weirdly over-represented in ML articles.

I work on these systems, and if anything my only complaint about the field is the propensity to solve every optimisation problem with ML. I have seen people solve textbook-grade linear, and even differentiable, optimisation problems. And the reason it happens despite the 'invisible hand' etc is because it still works, it just happens to be horrendously inefficient . I think that's the main area of inefficiency in the…

Fwiw, my org heavily relies on LP solvers in conjunction with ML to solve these problems

Re: Real World Recommendation System

#123
post #83
post #46

Earlier quoted context omitted.

Saying the article is "pretty shallow and non-descript: is much shorter and more useful than what they posted.

It provides more information. It's shallow and non-descript because it's an ad is the argument. I don't know if I believe that here. It's a blurry line with sponsored content.

Well then it's not "short-hand".

Re: Real World Recommendation System

#124
post #46

Earlier quoted context omitted.

Saying the article is "pretty shallow and non-descript: is much shorter and more useful than what they posted.

Right, but then it starts a meta-conversation about why the article got posted, or even written. It doesn't have the down-the-rabbit hole trait of an individual project of passion, or the sort of authoritative voice of a conference talk or even a Netflix blog post, it doesn't really speak to specific actionable technologies so it's not the kind of onboarding a Toward Data Science post would be. And that meta conversa…

> So just saying "this is content marketing" I think is a shibboleth for the entire conversation that starts with "pretty shallow and non-descript".

I'd disagree with this pretty strongly since there are many examples of content marketing that are also very useful pieces of content.

Re: Real World Recommendation System

#125
post #25

Earlier quoted context omitted.

(Disclaimer: I'm the author of the post) Good feedback, noted. Will get the next post focused on training within the next couple of days.

Also, how is this article different or more informative compared to others that deal with the challenges of model deployment/management at scale?

Model deployment is an important but still tiny part of the overall ranking/recommendation systems. Bulk of the complexity stems from two key properties of recommendation systems (which are different from say computer vision models): 1. The system operates on user feedback. As a result, it needs to manage flow of lots of data, with at least some subset being managed in realtime.

2. For any single request, there are thousands of things to recommend from. As a result, a single request is not scoring a single ML model but thousands of models - one (or often more, see value modeling the post) for each candidate.

Re: Real World Recommendation System

#126
post #51

Off-topic, but how did Netflix manage to get itself inserted into the FAANG acronym anyway? Their impact on the tech industry is trivial compared to all the others. Sure, if you just take out the N it's offensive, but we could have said "GAFA" or "FAAMG" would be more accurate to include Microsoft in their place.

[deleted]

Re: Real World Recommendation System

#127
post #105

Earlier quoted context omitted.

>Do you think part of this is that Netflix has assumed zero effort from user model? Talking w/a friend who works at Netflix, it sounds like this is a warranted assumption. The way he told it, they were tearing their hair out at one point b/c users wouldn't put much into it.

What I don't understand about their response is: why not make it configurable? Admittedly this is my philosophy for almost every product I work on - "make it maximally configurable, but make the defaults maximally sane" – but I'm baffled every time I hear someone talking about this 'dilemma'. You just keep your simple interface, but allow the power users to, say, click through to a particular menu and change their se…

I agree, but as aleksiy123 suggests there is an additional complexity burden and it is a long journey to teach users to make use of a new technology. I think a lot of "advanced" features get de-prioritized as not many people use them and it seems like resources could be better spent helping the masses. I think that the importance of "advanced" features is often under rated by traditional engagement models. Wikipedia is a great example of where less than 1% of users click on the edit button, but that 1% adds all the value for the other 99%.

Re: Real World Recommendation System

#128

> As a result, primary databases (e.g. MySQL, Mongo etc.) almost never work I mean it does. As far as I'm aware Facebook's ad platform is mostly backed by hundreds of thousands of Mysql instances. But more importantly this post really doesn't describe issues of scale. Sure it has the stages of recommendation, that might or might not be correct, but it doesn't describe how all of those processes are scheduled, coordin…

[deleted]

Re: Real World Recommendation System

#130
post #96

Earlier quoted context omitted.

TikTok seem to be learning from what the user is actually watching and for how long and not just the user's "Like"/"Not Interested In" actions. However it still seem to learn from the "Not Interested In" action more than any other platform.

This is a pretty misinformed take when it’s publicly known that YouTube was already doing this (learn from what the user is watching and for how long) the year Bytedance was founded (2012): https://blog.youtube/news-and-events/youtube-now-why-we-focu...

Somehow they're doing it better. At least subjectively, people complain more about the YouTube algo's performance than tiktok. For the latter, the most common complaint is that it's too good.
Post reply on HN