Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix's Metaflow: Reproducible machine learning pipelines

cortex.dev

Post reply on HN
Print view View original

61–70 of 105 posts

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#61

Post by jaz46 »

the_duke wrote: post #20

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experi…

There are lots of great platforms and tools in this space that are trying to solve these problems. They all have their tradeoffs and of course the list of needs/goals above is pretty diverse.

You are very likely going to be using a handful of tools that cover to full gamut of needs. This is heavily discussed in the blog post about an MLOps Canonical Stack and many of the tools being suggested below are included.

https://towardsdatascience.com/rise-of-the-canonical-stack-i...

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#62

Post by merkleforest »

itamarst wrote: post #19
vtuulos wrote: post #5

If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

Hey, been meaning to reach out. There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely." The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expen…

It would be great if the infra layer can provide some help on automated resource scaling, especially for RAM. The ML solver/tooling layer has also been making progress on this front, for example Dask for limited-RAM pandas, h2o.ai has limited RAM solvers, xgboost has an external memory version, pytorch/tensorflow models are mostly trained on SGD and only needs to load data batch by batch. It's nice that Metaflow can integrate with any python code and thus benefit from all of the efforts made on the solver/tooling layer.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#63

Post by ramraj07 »

DoofusOfDeath wrote: post #9
ed25519FUUU wrote: post #3

Earlier quoted context omitted.

Do you have a counter example of a streaming service that does recommendations better?

Back when I used Netflix primarily for DVDs, the recommender system worked pretty well for me. Much later, when they switched to simple thumbs up/down, the recommender system was entirely useless to me. (Not merely because of the dumbed-down rating system; the recommendations were genuinely bad.) For the time in between, I'm not sure if the degradation was gradual, sporadic, or not degraded at all.

If you add dvd subscriptions you can still get the original Netflix recommendations back, including sorting by top predicted rating (which to me is eerily amazing). I used to keep the dvd subscription mainly for the recommender, with the delivered blurays considered an extra bonus for very rare movies you couldn't stream even if you wanted to pay.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#64

Post by dddbbb »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Maybe it's different in other countries (I am in the UK), but I feel as if there isn't enough content for a recommender system to even be useful. I feel like after browsing through the catalogue a few times, I have a rough idea of most things I would ever possibly be interested in. There's just not that much there. Either that, or the recommender system is working too well and I never see anything beyond what Netflix wants me to.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#65

Post by jointpdf »

octostone wrote: post #54
jointpdf wrote: post #39

Earlier quoted context omitted.

It’s not a home run for me, but I think Spotify recommendations are quite good. They clearly use some form of content based recommendation (extracting features from the music itself) blended with other methods. It seems to make an honest attempt at serendipity (songs/artists you may like but would otherwise be unlikely to discover). I still think recommender engines should always enable some form of user tuning. If i…

Unless I’m missing this functionality somewhere (entirely possible), this lack of user-tuning has ruined Spotify recs for me, since I listen to entirely different playlists when working or meditating. I don’t check out recommendations to get the latest binaural beats or nature sounds, you know? Although even before I started listening to Spotify while working etc, it seemed to have run out of things to recommend. My…

I personally have a similar problem with the Spotify generated playlists, but I have persnickety preferences in electronic music so it often whiffs. But on the spectrum of recommendation engines it is on the side of honest effort (whereas Netflix is not). And for most users and musical palates I think it works great. Just yesterday my mom complimented me on my Christmas music DJ skills, but it was just the generated continuation of her own playlist.

Nothing really beats the recommendations of a human curator with exquisite taste, and just listening to new music nonstop and plucking out the gems as you go.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#66

Post by toxik »

alisonkisk wrote: post #36
hateful wrote: post #18

Earlier quoted context omitted.

I really think that if they gave up all the complicated algorithms and went with a simple algorithm out of the 90s we'd be much happier with the recommendations.

Do you have such an algorithm? Netflix doesn't make money by showing people stuff they don't like.

Not for a lack of trying though...

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#67

Post by toxik »

ed25519FUUU wrote: post #3
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Do you have a counter example of a streaming service that does recommendations better?

The basic problem is understanding WHY I liked something. If I watch Tintin because it’s a cozy throwback to my childhood, that does not mean I would like every single Studio Ghibli movie in my recommendations.

Similarly, if I play Blacklist in the background as basically noise, I don’t want to see a bunch of related shows. I guess I could give it a thumbs down but I only do that for actually terrible movies.

Also Spotify and Apple music seem to have okay recommendations

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#68

Post by kyawzazaw »

visarga wrote: post #8
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Not just the recommender, but the design is horrible for me. I can't rest my mouse anywhere, it auto-starts something or enlarges something, it's all too twitchy. When you're watching an episode, there's no navigation link from the play screen to the main page of the series. As if they don't want us to navigate the site, instead be led on their happy path.

I think so. This way, you don't easily arrive to the conclusion the items I want to look for all don't exist here.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#69

Post by kyawzazaw »

toxik wrote: post #67
ed25519FUUU wrote: post #3

Earlier quoted context omitted.

Do you have a counter example of a streaming service that does recommendations better?

The basic problem is understanding WHY I liked something. If I watch Tintin because it’s a cozy throwback to my childhood, that does not mean I would like every single Studio Ghibli movie in my recommendations. Similarly, if I play Blacklist in the background as basically noise, I don’t want to see a bunch of related shows. I guess I could give it a thumbs down but I only do that for actually terrible movies. Also Sp…

Music is different from movies and television though. There are beats and rhythms that are easy to identify, lyrics easy to analyze and artists that are roughly categorized.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#70

Post by nerdponx »

alisonkisk wrote: post #34
drstewart wrote: post #17

Earlier quoted context omitted.

>Finally, they replaced the ratings with “% matching”. I guess so they can recommend their subpar productions even if they get low ratings. That's always what ratings were. People didn't understand that (as you can see), so they changed it to make it more transparent. https://www.businessinsider.com/why-netflix-replaced-its-5-s... >Netflix’s star ratings were personalized, and had been from the start. That means when…

It's not just that (obviously, since It's not even possible for me to rate a movie more granular than thumbs up/down) What happened is that the notion changed from "predict a scalar rating" to "predict a binary satisfaction." As parent poster noted, the effect of this is to push "3 star" and "4 star" acceptable shows to the user, instead of "5 star" great (in the user's view) shows. Also, in Netflix's defense, users…

Also, in Netflix's defense, users are horribly inconsistent in their expressed ratings (they'll rate a movie based on their personal mood at the time, and they'll binge shows they claim aren't 5 stars while ignoring their 5 star movies)

This is exactly why they switched from 5 stars to up/down/blank.

Post reply on HN