Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix's Metaflow: Reproducible machine learning pipelines

cortex.dev

Post reply on HN
Print view View original

21–30 of 105 posts

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#21

Post by umvi »

Hard_Space wrote: post #4
toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

I long ago ceased to believe that the Netflix recommender system serves any other purpose than to fulfill the company's internal obligations to push favored content, depending on what it cost. Sadly, the same is now true for Amazon Prime, which is an even hotter mess.

I turned on "Super Wings" for my kid to watch on Prime Video which at first glance seemed to be a fairly decent Paw Patrol knock off, but then as I listened to the episodes in the background, I realized that the entire show is basically an advertisement for Amazon Prime in disguise. Seriously look it up, the entire premise of the show is people ordering packages and the "Super Wings" delivering the packages to the consumer as quickly as possible... .

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#22

Post by [deleted] »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

[deleted]

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#24

Post by [deleted] »

itamarst wrote: post #19
vtuulos wrote: post #5

If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

Hey, been meaning to reach out. There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely." The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expen…

[deleted]

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#25

Post by manojlds »

savin-goyal wrote: post #15
pknerd wrote: post #14

I am trying to figure out Kubflow. Surprisingly I found this one easier to write. Haven't run it or used it yet.

Take a look at metaflow.org/sandbox if you want to test drive Metaflow.

It takes me into a verification and waiting flow. Useless.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#26

Post by vtuulos »

itamarst wrote: post #19
vtuulos wrote: post #5

If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

Hey, been meaning to reach out. There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely." The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expen…

I'd love to hear more what you have in mind! Feel free to drop by at our chat at https://gitter.im/metaflow_org/community

While it is true that auto-sizing resources is hard and the easiest approach is to oversize @resources, the situation isn't as bad as it sounds:

1) In Metaflow, @resource requests are specific to a function/step, so you end up using resources only for a short while typically. It would be expensive to keep big boxes idling 24/7 but that's not necessary.

2) You can use spot instances to lower costs, sometimes dramatically.

3) It is pretty easy to see the actual resource consumption on any monitoring system, e.g. CloudWatch, so you can adjust manually if needed.

4) A core value proposition of Metaflow is to make both prototyping and production easy. While optimizing resource consumption may be important for large-scale production workloads, it is rarely the first concern when prototyping.

In practice at Netflix, we start with overprovisioning and then focus on optimizing only the workflows that mature to serious production and end up being too expensive if left unoptimized. It turns out that this is a small % of all workflows.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#27

Post by mmq »

the_duke wrote: post #20

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experi…

Hi @the_duke,

disclaimer I am one of the authors of an open-source solution (https://github.com/polyaxon/polyaxon) that specializes in the experimentation and automation phase of the data-science lifecycle.

Our tool provides exactly the kind of abstraction you mentioned:

* Training, data operations, and interactive workspaces (https://polyaxon.com/docs/experimentation/)

* A scalable history and comparison table (https://polyaxon.com/docs/management/runs-dashboard/comparis...)

* Currently pipelines and concurrency management is on the commercial version (https://polyaxon.com/docs/automation/) but several companies use Polyaxon with other tools like Kubeflow (https://medium.com/mercari-engineering/continuous-delivery-a...) or it can be used with MetaFlow for the pipelines part.

I would really like to hear your thoughts and feedback.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#28

Post by vtuulos »

smurf_t wrote: post #23

What are the main advantages compared to Airflow. We use Airflow to orchestrate ML jobs/tasks, and I found it to be more flexible comparing other tools we tested.

Metaflow is largely complementary to a job scheduler like Airflow. Technically you could export Metaflow workflows to Airflow, although the specific integration doesn't exist yet. For more details, see this blog article https://netflixtechblog.com/unbundling-data-science-workflow...

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#29

Post by troelsSteegin »

vtuulos wrote: post #5

If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

What is Metaflow's explicit support for Transfer Learning tasks? In other words, how do I know what models to use or not use? I am surmising from the techblog post that there is a stable set of content-intrinsic features, and that can be separated from perhaps more dynamic features-sets that characterize audiences, presentation treatment, and viewing (as conditioned on all the other stuff). But it sounds like there is a stable set of features for prediction tasks, as well, which is to say that for a task like predicting an audience for movie X in region Y, you'll need some set of features, and that we have some set of trained models (and recommend analytic components) available that match some or all of those features for this task. Is that a "thing", or is the workflow support simpler than that, and should that be a "thing"?

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#30

Post by solumos »

the_duke wrote: post #20

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experi…

Same - I'm a SWE embedded in a small (but growing) ML team. We have all of the same problems.

It seems that the "all-in" platforms are too "rigid", and all of the point solutions for the things you mentioned aren't proven enough.

Post reply on HN