Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix's Metaflow: Reproducible machine learning pipelines

cortex.dev

Post reply on HN
Print view View original

31–40 of 105 posts

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#31

Post by Reubend »

ed25519FUUU wrote: post #3
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Do you have a counter example of a streaming service that does recommendations better?

The MovieLens recommender system works pretty well.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#32

Post by savin-goyal »

the_duke wrote: post #20

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experi…

Hi! Metaflow ships with a CloudFormation template for AWS that automates the set-up of a blob store (S3), compute environment (Batch), metadata tracking service (RDS), orchestrator (Step-Functions) notebooks (Sagemaker) and all the necessary IAM permissions to ensure data integrity. Using Metaflow, you can then write your workflows in Python/R and Metaflow will take care of managing your ML dev/prod lifecycle.

https://github.com/Netflix/metaflow-tools/tree/master/aws/cl...

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#34

Post by alisonkisk »

drstewart wrote: post #17
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

>Finally, they replaced the ratings with “% matching”. I guess so they can recommend their subpar productions even if they get low ratings. That's always what ratings were. People didn't understand that (as you can see), so they changed it to make it more transparent. https://www.businessinsider.com/why-netflix-replaced-its-5-s... >Netflix’s star ratings were personalized, and had been from the start. That means when…

It's not just that (obviously, since It's not even possible for me to rate a movie more granular than thumbs up/down)

What happened is that the notion changed from "predict a scalar rating" to "predict a binary satisfaction."

As parent poster noted, the effect of this is to push "3 star" and "4 star" acceptable shows to the user, instead of "5 star" great (in the user's view) shows.

Also, in Netflix's defense, users are horribly inconsistent in their expressed ratings (they'll rate a movie based on their personal mood at the time, and they'll binge shows they claim aren't 5 stars while ignoring their 5 star movies)

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#35

Post by alisonkisk »

umvi wrote: post #21
Hard_Space wrote: post #4

Earlier quoted context omitted.

I long ago ceased to believe that the Netflix recommender system serves any other purpose than to fulfill the company's internal obligations to push favored content, depending on what it cost. Sadly, the same is now true for Amazon Prime, which is an even hotter mess.

I turned on "Super Wings" for my kid to watch on Prime Video which at first glance seemed to be a fairly decent Paw Patrol knock off, but then as I listened to the episodes in the background, I realized that the entire show is basically an advertisement for Amazon Prime in disguise. Seriously look it up, the entire premise of the show is people ordering packages and the "Super Wings" delivering the packages to the co…

Seems like you got what you expected... Super Wings is the corporate side of the Government-Corporate complex.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#36

Post by alisonkisk »

hateful wrote: post #18
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

I really think that if they gave up all the complicated algorithms and went with a simple algorithm out of the 90s we'd be much happier with the recommendations.

Do you have such an algorithm?

Netflix doesn't make money by showing people stuff they don't like.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#38

Post by thecellardoor »

solumos wrote: post #30
the_duke wrote: post #20

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experi…

Same - I'm a SWE embedded in a small (but growing) ML team. We have all of the same problems. It seems that the "all-in" platforms are too "rigid", and all of the point solutions for the things you mentioned aren't proven enough.

I think that by definition this is a tradeoff. Most times you talk to data scientists that want a fully automated end-2-end solution, that doesn't require they change anything about their current workflow, and that any future modifications to their workflow would be supported as well.

That is magical thinking. I prefer best of breed solutions that integrate nicely with other best of breed solutions every day. That way if a tool doesn't suit you tomorrow, you can relatively easily swap it out for something better

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#39

Post by jointpdf »

ed25519FUUU wrote: post #3
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Do you have a counter example of a streaming service that does recommendations better?

It’s not a home run for me, but I think Spotify recommendations are quite good. They clearly use some form of content based recommendation (extracting features from the music itself) blended with other methods. It seems to make an honest attempt at serendipity (songs/artists you may like but would otherwise be unlikely to discover).

I still think recommender engines should always enable some form of user tuning. If it doesn’t, then the recommender is a tool for services to control your behavior rather than the other way around.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#40

Post by edameme »

the_duke wrote: post #20

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experi…

I'd love to hear what you think about https://dagshub.com/. We're building it with community collaboration in mind. It doesn't cover all the bases you mention, but we do: * data and model storage * experiment tracking * pipeline management * access control * data, model, code, pipeline versioning

We're also strictly based on Git and other Open Source formats and tools so connecting with other tools you use like Colab for IDEs or Jenkins/Kubeflow for training is super straightforward (we have examples for some)

Post reply on HN