Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix's Metaflow: Reproducible machine learning pipelines

cortex.dev

Post reply on HN
Print view View original

71–80 of 105 posts

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#71

Post by fwip »

itamarst wrote: post #19
vtuulos wrote: post #5

If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

Hey, been meaning to reach out. There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely." The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expen…

One way I deal with this problem in an alternative workflow manager (Nextflow), is by calculating the memory requirement for the ~95th percentile of a job, and submitting with a rule "If this crashes from going OOM, re-submit with memory*N" (up to some max number of retries/RAM). This lets most jobs sail through with a relatively low amount of RAM, and the bigger jobs end up taking a bit more time and resources.

The better your estimator function, of course, the tighter constraints you can use.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#72

Post by toxik »

kyawzazaw wrote: post #69
toxik wrote: post #67

Earlier quoted context omitted.

The basic problem is understanding WHY I liked something. If I watch Tintin because it’s a cozy throwback to my childhood, that does not mean I would like every single Studio Ghibli movie in my recommendations. Similarly, if I play Blacklist in the background as basically noise, I don’t want to see a bunch of related shows. I guess I could give it a thumbs down but I only do that for actually terrible movies. Also Sp…

Music is different from movies and television though. There are beats and rhythms that are easy to identify, lyrics easy to analyze and artists that are roughly categorized.

I feel like that’s not true, music is hard to analyze. Movies have synopses, and are generally categorized by their cast alone. I could figure out LotR belongs together from these datapoints, easily. Not to mention the fact that people watch trilogies in sequence like 99% of the time.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#73

Post by phillc73 »

dddbbb wrote: post #64
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Maybe it's different in other countries (I am in the UK), but I feel as if there isn't enough content for a recommender system to even be useful. I feel like after browsing through the catalogue a few times, I have a rough idea of most things I would ever possibly be interested in. There's just not that much there. Either that, or the recommender system is working too well and I never see anything beyond what Netflix…

I have to agree. As far as I recall the Netflix search engine has never found the exact title I've been searching for. Even the recommendations it then shows me are so far off piste that they're not really even close to what I'm looking for.

At least with Amazon Prime there's a high chance they'll at least find the title one searches for and if I am really motivated I can pay a few bucks to watch it.

Netflix just draws a blank.

That's not to say I haven't watched some entertaining things on Netflix, but they seem much better suited to TV series than movies, and I almost feel like I found decent things to watch inspite of their recommendation system, not because of it.

An ML tool from Netflix? It feels like the last thing I'm likely to use.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#74

Post by freetonik »

visarga wrote: post #8
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Not just the recommender, but the design is horrible for me. I can't rest my mouse anywhere, it auto-starts something or enlarges something, it's all too twitchy. When you're watching an episode, there's no navigation link from the play screen to the main page of the series. As if they don't want us to navigate the site, instead be led on their happy path.

Good news is that you can disable autoplay of previews [0]. Bad news is that it takes weeks to propogate this setting change to all clients. I had to wait about two weeks for my apply tv client to stop auto previews. I suspect the queue service is powered by snail mail.

[0] https://help.netflix.com/en/node/2102

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#75

Post by johncena33 »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Can we please take these off-topic rants somewhere else? Lately, it's hard to find any insightful comments in the disucssion section of engineering or technical articles. Because the whole conversation is derailed by something unrelated off-topic rants. There are a lot of social media, forums and even HN submissions where everyone can rant. Please keep the technical submissions clean.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#76

Post by aeoanx »

freetonik wrote: post #74
visarga wrote: post #8

Earlier quoted context omitted.

Not just the recommender, but the design is horrible for me. I can't rest my mouse anywhere, it auto-starts something or enlarges something, it's all too twitchy. When you're watching an episode, there's no navigation link from the play screen to the main page of the series. As if they don't want us to navigate the site, instead be led on their happy path.

Good news is that you can disable autoplay of previews [0]. Bad news is that it takes weeks to propogate this setting change to all clients. I had to wait about two weeks for my apply tv client to stop auto previews. I suspect the queue service is powered by snail mail. [0] https://help.netflix.com/en/node/2102

That must be because of an excessive amount of caching on the client-side

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#77

Post by ricklamers »

the_duke wrote: post #20

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experi…

Others have mentioned some cool projects in this space, but you mentioned self hosted specifically so I’ll share what we’re working on since it might match what you’re looking for.

As a new project we are still figuring out some of major topics you described.

In short, we built a data science pipeline tool that should fit well with existing workflows in machine learning and data science. We chose to embrace and integrate open source projects to create a simple and seamless experience with best in breed solutions for various tasks.

We are particularly happy with our deep integration of JupyterLab building on the excellent Jupyter Enterpise Gateway project from IBM (Codait) for connecting kernels directly to your pipelines. For scheduling we build on top of Celery combined with containerization primitives. For stable and well defined dependency management we built a small environment abstraction on top of Docker. It works really well in our experience!

Feel free to check out the project on https://github.com/orchest/orchest

Self hosting should be as easy as running about two lines of code.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#78

Post by itamarst »

fwip wrote: post #71
itamarst wrote: post #19

Earlier quoted context omitted.

Hey, been meaning to reach out. There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely." The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expen…

One way I deal with this problem in an alternative workflow manager (Nextflow), is by calculating the memory requirement for the ~95th percentile of a job, and submitting with a rule "If this crashes from going OOM, re-submit with memory*N" (up to some max number of retries/RAM). This lets most jobs sail through with a relatively low amount of RAM, and the bigger jobs end up taking a bit more time and resources. The…

In many cases the memory usage is linear with input data, so you can come up with a function that predicts memory usage, add some padding, and then you don't need retries. E.g. this example here: https://pythonspeed.com/articles/estimating-memory-usage/

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#79

Post by tsjq »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Fully agree with this. Yuck, Netflix recos are ultra bad.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#80

Post by Havoc »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

>Netflix’s recommender system is hands down the worst

Until you log into prime video.

Can’t manage to give me a “continue watching last thing button”. That’s literally the most likely thing I want to watch. Also routinely suggest starting with S02 even though I’ve not watched S01.

Never mind machine learning some common sense would be greatly appreciated

Post reply on HN