Netflix's Metaflow: Reproducible machine learning pipelines • Hacker News Board

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#101

Post by Havoc » Tue, Dec 22, 2020, 1:34 PM UTC

bra-ket wrote: post #88

Havoc wrote: post #80

Earlier quoted context omitted.

>Netflix’s recommender system is hands down the worst Until you log into prime video. Can’t manage to give me a “continue watching last thing button”. That’s literally the most likely thing I want to watch. Also routinely suggest starting with S02 even though I’ve not watched S01. Never mind machine learning some common sense would be greatly appreciated

Netflix actually organized a major machine learning competition more than a decade ago[1] with thousands of the best researchers trying to beat some internal benchmark by a few percent, for $1M prize, I'm wondering where did all that "learning" go. My guess is that whatever rankings they currently produce maximize some internal revenue target and that target' user base is not me or you. [1] https://en.wikipedia.org/w…

Yup. It just ends up irritating users though if the suggestions break basic common sense.

If I watched S01E04 yesterday I want to watch S01E05 today. The interface should be suggesting that PLUS whatever else the ML comes up with in addition, not instead of.

Prime is full of minor irritations like that make me wonder whether Amazon engineers dogfood enough

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#102

Post by nicktorba » Tue, Dec 22, 2020, 3:39 PM UTC

I really like the combination of these two tools.

I've played with cortex before, and it is easy to use, but I am still questionable if automating kubernetes deployments through an easy code interface, without much kubernetes know-how, is safe.

In my experience, even when you have a tool automating a lot of kubernetes for you, you will still run into trouble that will be best handled if you are familiar with kubernetes. I'm not sure what debugging utilities cortex has, but I think the ultimate solution to this problem will be a tool that truly allows users to not think about the fact their deployments are running on kubernetes at all.

I'm also interested in the similarities of Cortex and Seldon-core. Of course, seldon-core does not automate infra provisioning, but based on my previous point, I think many teams are better off being more hands on with this infra.

Lastly, there is a third tool missing from the mix - monitoring. I think cortex offers some tools in this area, but I wish they would make a part two showing how the monitoring functionality they offer can integrate into a retraining pipeline within metaflow. This post shows you how to get started, but it doesn't show you how to maintain applications long term.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#103

Post by vtuulos » Tue, Dec 22, 2020, 8:14 PM UTC

jlkim wrote: post #99

vtuulos wrote: post #5

If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

Are there any built-in dashboards for such actions as querying or analysing model versions and the metadata around them?

Coming soon!

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#104

Post by nicktorba » Wed, Dec 23, 2020, 4:18 AM UTC

sandGorgon wrote: post #91

has anyone done a comparison of ML pipelines from a devops centric perspective ? For example, Metaflow doesnt support kubernetes today - https://github.com/Netflix/metaflow/issues/16 so ultimately the scale up story in most of these management tools is iffy. I previously asked about kubeflow here - https://news.ycombinator.com/item?id=24808090 . Seems people think its pretty "horrendous". It seems most of these tools…

This would be super useful.

Based on this thread, the comparison should include

* metaflow (model training on AWS Batch) * polyaxon (model training on kubernetes) * pachyderm (experimentation) * hopsworks (model training/serving/ and more, mostly on kubernetes) * cortex (model serving on kubernetes) * seldon-core (model serving and monitoring on kubernetes)

and likely more that I missed.

I can see why it would be so hard to put together this comparison.

Even with all these tools, there is still a lot of manual work for data scientists or DevOps engineers the data scientists pass their models off to.

It also seems there is yet to be a fully open source DevOps stack. Most companies still build custom software to glue together manual processes (like integrations between different tools for training, deploying, monitoring, etc). This could be one factor why more comparisons of these tools and stack discussions have not been more popular - they can't share them yet.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#105

Post by nextaccountic » Wed, Dec 23, 2020, 4:38 AM UTC

vtuulos wrote: post #89

nextaccountic wrote: post #87

Earlier quoted context omitted.

Hi! How do you handle floating point determinism? Can some ML be reproduced in any architecture? Can you build the code with another compiler version? Can you use newer SIMD instructions? Or you're forever tied to the initial hardware+compiler version?

Reproducibility is a spectrum. A good starting point is to snapshot the exact version of the code that produced a model. Even better, you should snapshot the full dependency graph, including transitive dependencies, of all (compiled) libraries, which Metaflow does with @conda. Together with data snapshots, this gives a pretty good foundation for reproducibility. Depending on the libraries you use, the exact results m…

Is there any library in the metaflow ecosystem that offers floating point determinism? I would like to read more, can you link somewhere?

Also, what tools does metaflow offers to control the level of reproducibility?