Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix's Metaflow: Reproducible machine learning pipelines

cortex.dev

Post reply on HN
Print view View original

11–20 of 105 posts

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#11

Post by dessant »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Netflix often promotes new content by overzealously recommending it, the constant suggestion you see for The Queen’s Gambit is probably an ad.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#12

Post by qw3rty01 »

ed25519FUUU wrote: post #3
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Do you have a counter example of a streaming service that does recommendations better?

Not a streaming service, but a lot of movie and TV databases tend to have decent recommendations for movies and shows similar to the one you're viewing. Although in this case, one could argue that the processing is offloaded to users who provide the recommendations.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#13

Post by typon »

emidln wrote: post #10
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Their recommender system likely has multiple inputs that are under specific constraints and weightings. How well their recommendation algorithms work will probably never be known by the general public as we are force-fed a steady diet of whitelisted staff picks and promotional items.

So what's the point of making tools like Metaflow? Justifying salaries for their engineers I suppose?

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#16

Post by treis »

DoofusOfDeath wrote: post #9
ed25519FUUU wrote: post #3

Earlier quoted context omitted.

Do you have a counter example of a streaming service that does recommendations better?

Back when I used Netflix primarily for DVDs, the recommender system worked pretty well for me. Much later, when they switched to simple thumbs up/down, the recommender system was entirely useless to me. (Not merely because of the dumbed-down rating system; the recommendations were genuinely bad.) For the time in between, I'm not sure if the degradation was gradual, sporadic, or not degraded at all.

It's pretty clear that their recommendation system and broader UX is designed, at least in part, to obfuscate how much content they have and how good it is. Back when it was DVDs and they had basically everything it was more about finding the next best thing for you. Now it's finding the next best thing they have and, preferably, something they own the rights to.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#17

Post by drstewart »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

>Finally, they replaced the ratings with “% matching”. I guess so they can recommend their subpar productions even if they get low ratings.

That's always what ratings were. People didn't understand that (as you can see), so they changed it to make it more transparent.

https://www.businessinsider.com/why-netflix-replaced-its-5-s...

>Netflix’s star ratings were personalized, and had been from the start. That means when you saw a movie on Netflix rated 4 stars, that didn’t mean the average of all ratings was 4 stars. Instead, it meant that Netflix thought you’d rate the movie 4 stars, based on your habits (and other people's ratings). But many people didn’t get that.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#18

Post by hateful »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

I really think that if they gave up all the complicated algorithms and went with a simple algorithm out of the 90s we'd be much happier with the recommendations.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#19

Post by itamarst »

vtuulos wrote: post #5

If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

Hey, been meaning to reach out.

There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely."

The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expensive at scale.

This choice between "my process crashes" and "I am incentivized to make my process organizationally expensive" isn't ideal. Do you have any ways you deal with this at Netflix, or have you seen ways other Metaflow users deal with it?

I have some ideas on how this could be made better (some combination of being able to catch OOM situations deterministically, memory profiling, and sizing RAM by input size for repeating batch jobs), based in part on some tooling I've been working on for memory profiling: https://pythonspeed.com/fil, so would love to talk about it if you're interested.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#20

Post by the_duke »

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult.

What I really want is a single solution, or a set of pluggable, integrated components that offer:

* training data and model storage (on top of a blob store like S3, minio, ...)

* interactive dev environments (Notebooks, dev containers, ...)

* training (with history, comparisons, parameters, ...) with experiments for parameter tuning

* serving/deploying for production

* a permission system so researchers and developers can only access what they are supposed to

* software heritage, probably via Docker images or Nix packages, combined with source code references

* (cherry on top: some kind of integrated labeling system and UI)

Right now you have to cobble this together from different tools that are all pretty suboptimal.

The big players can set up sophisticated systems, but I'm curious to hear how other startups are currently solving this.

Post reply on HN