Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix's Metaflow: Reproducible machine learning pipelines

cortex.dev

Post reply on HN
Print view View original

81–90 of 105 posts

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#81

Post by JohnJamesRambo »

ed25519FUUU wrote: post #3
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Do you have a counter example of a streaming service that does recommendations better?

Spotify's Discover Weekly has a pipeline straight into my brain and not only what I like, but what I will like in the future.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#82

Post by zaroth »

aeoanx wrote: post #76
freetonik wrote: post #74

Earlier quoted context omitted.

Good news is that you can disable autoplay of previews [0]. Bad news is that it takes weeks to propogate this setting change to all clients. I had to wait about two weeks for my apply tv client to stop auto previews. I suspect the queue service is powered by snail mail. [0] https://help.netflix.com/en/node/2102

That must be because of an excessive amount of caching on the client-side

You would hope that logging out and back in would refresh these settings.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#83

Post by zaroth »

the_duke wrote: post #20

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experi…

Just wanted to let you know I favorited this comment for the responses it garnered. Great question, great efforts at providing answers.

This could be its own “Ask HN” thread.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#84

Post by mr_toad »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

> Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions.

I always assumed those three were human selections, not part of the recommendations.

Like the paid ads on top of your search results.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#86

Post by 1123581321 »

AnssiH wrote: post #42
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

I see this complaint about poor recommendations very often. But the recommendations seem to work perfectly for me - I wonder whether that is the case for the silent majority? The match % is usually spot on for me, and I've never seen it recommend any titles I've given a thumbs-down to.

For many of the complainers, Netflix is showing them something they would actually enjoy. How to communicate that is challenging. I remember Pandora’s generated stations having a similar issue years ago, where it would catch similarities between on-trend bands and bands that had fallen out of favor. It was right, but it could sometimes be hard to submit to the insight of the algorithm. Spotify is one that I think does this well as they understand how not to give a recommendation that insults or offends, even if it’s technically an accurate response.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#87

Post by nextaccountic »

vtuulos wrote: post #5

If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

Hi! How do you handle floating point determinism? Can some ML be reproduced in any architecture? Can you build the code with another compiler version? Can you use newer SIMD instructions?

Or you're forever tied to the initial hardware+compiler version?

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#88

Post by bra-ket »

Havoc wrote: post #80
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

>Netflix’s recommender system is hands down the worst Until you log into prime video. Can’t manage to give me a “continue watching last thing button”. That’s literally the most likely thing I want to watch. Also routinely suggest starting with S02 even though I’ve not watched S01. Never mind machine learning some common sense would be greatly appreciated

Netflix actually organized a major machine learning competition more than a decade ago[1] with thousands of the best researchers trying to beat some internal benchmark by a few percent, for $1M prize, I'm wondering where did all that "learning" go.

My guess is that whatever rankings they currently produce maximize some internal revenue target and that target' user base is not me or you.

[1] https://en.wikipedia.org/wiki/Netflix_Prize

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#89

Post by vtuulos »

nextaccountic wrote: post #87
vtuulos wrote: post #5

If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

Hi! How do you handle floating point determinism? Can some ML be reproduced in any architecture? Can you build the code with another compiler version? Can you use newer SIMD instructions? Or you're forever tied to the initial hardware+compiler version?

Reproducibility is a spectrum. A good starting point is to snapshot the exact version of the code that produced a model. Even better, you should snapshot the full dependency graph, including transitive dependencies, of all (compiled) libraries, which Metaflow does with @conda. Together with data snapshots, this gives a pretty good foundation for reproducibility.

Depending on the libraries you use, the exact results may or may not be reproducible on other architectures. If cross-platform reproducibility is important to you, you should choose your libraries accordingly. Metaflow provides the tools for choosing the level of reproducibility that your application requires.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#90

Post by manojlds »

mmq wrote: post #27
the_duke wrote: post #20

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experi…

Hi @the_duke, disclaimer I am one of the authors of an open-source solution ( https://github.com/polyaxon/polyaxon ) that specializes in the experimentation and automation phase of the data-science lifecycle. Our tool provides exactly the kind of abstraction you mentioned: * Training, data operations, and interactive workspaces ( https://polyaxon.com/docs/experimentation/ ) * A scalable history and comparison table (…

> but several companies use Polyaxon with other tools like Kubeflow or it can be used with MetaFlow for the pipelines part.

Isn't that what the parent you are replying to is talking about with "Right now you have to cobble this together from different tools that are all pretty suboptimal."

Post reply on HN