Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Netflix's Metaflow: Reproducible machine learning pipelines

cortex.dev

Post reply on HN
Print view View original

91–100 of 105 posts

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#91

Post by sandGorgon »

has anyone done a comparison of ML pipelines from a devops centric perspective ?

For example, Metaflow doesnt support kubernetes today - https://github.com/Netflix/metaflow/issues/16

so ultimately the scale up story in most of these management tools is iffy.

I previously asked about kubeflow here - https://news.ycombinator.com/item?id=24808090 . Seems people think its pretty "horrendous". It seems most of these tools assume a very specialised devops team who will work around the ml tool...rather than the ml tool making this easy.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#92

Post by vtuulos »

sandGorgon wrote: post #91

has anyone done a comparison of ML pipelines from a devops centric perspective ? For example, Metaflow doesnt support kubernetes today - https://github.com/Netflix/metaflow/issues/16 so ultimately the scale up story in most of these management tools is iffy. I previously asked about kubeflow here - https://news.ycombinator.com/item?id=24808090 . Seems people think its pretty "horrendous". It seems most of these tools…

As of today, Metaflow supports AWS Batch for scaling out and up. While supporting K8S is convenient in the operations point of view (assuming you have a K8S cluster already), it doesn't make the scalability story any better.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#94

Post by danielscrubs »

alisonkisk wrote: post #36
hateful wrote: post #18

Earlier quoted context omitted.

I really think that if they gave up all the complicated algorithms and went with a simple algorithm out of the 90s we'd be much happier with the recommendations.

Do you have such an algorithm? Netflix doesn't make money by showing people stuff they don't like.

They actually might... think of it like gym memberships. Sure you need a hook, but as Mandalorian showed, that can be a single exclusive tv show that doesn’t require much broadband cost. Maybe not the best solution long term but since when has that been the shareholders goal?

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#95

Post by sandGorgon »

vtuulos wrote: post #92
sandGorgon wrote: post #91

has anyone done a comparison of ML pipelines from a devops centric perspective ? For example, Metaflow doesnt support kubernetes today - https://github.com/Netflix/metaflow/issues/16 so ultimately the scale up story in most of these management tools is iffy. I previously asked about kubeflow here - https://news.ycombinator.com/item?id=24808090 . Seems people think its pretty "horrendous". It seems most of these tools…

As of today, Metaflow supports AWS Batch for scaling out and up. While supporting K8S is convenient in the operations point of view (assuming you have a K8S cluster already), it doesn't make the scalability story any better.

sure - im not talking about scalability from a science perspective, but really from practical applicability perspective.

Most people already are running k8s of some kind. I beginning to see k8s increasingly as an invariant. Plus not all of us are on AWS.

Secondly, AWS Batch is only applicable for metaflow for ml training.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#96

Post by ruksi »

the_duke wrote: post #20

Setting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experi…

we at Valohai (https://valohai.com/) do check all the mentioned feature boxes and do serve a lot of startups

* data stores: automatic download/upload to/from AWS S3, Azure Blob, Google Cloud Storage, OpenStack Swift or stores that implements S3-like interface

* interactive environments: we do have notebook hosting with automatic orchestration

* training: history, comparisons, parameters, hyperparameter tuning with Optuna, Hyperopt or custom optimizer (https://github.com/valohai/optimo); additionally visualizations about training progress and hardware resource monitoring

* serving for production: our deployments allow you to build, push, manage and monitor HTTP/S based services on Kubernetes clusters (https://docs.valohai.com/core-concepts/deployments/) but you can just as easily download your model and deploy it yourself as your use-case requires

* a permission system: we have organization management with teams and such, but your mileage may vary depending how fine grained control you need

* software heritage: all runs are containerized and how they was ran is recorded so everything is reproducible if the base image and data exist at the original source, we also keep track of data heritage (what files X were used to produce these files Y https://valohai.com/patch-notes/2019-09-03/)

* labeling system/UI: full web UI, command line client and a REST API (https://docs.valohai.com/valohai-api/) but no labeling tools though

Essentially your whole machine learning pipeline under one roof; from data preprocessing and training to deployment and monitoring. Also, we are technically agnostic, you can just as easily run Python/Julia/C++ or Unity engine to generate synthetic datasets (https://www.youtube.com/watch?v=QxMuWuk_W10)

not self-service or free though; our technical support team handles all the setup and maintenance

let me know if you have questions about Valohai or MLOps (https://valohai.com/mlops/) in general, I've seen quite a lot of projects and pipelines as I work at Valohai as an ML engineer helping our customers to setup end-to-end ML pipelines

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#97

Post by mmq »

manojlds wrote: post #90
mmq wrote: post #27

Earlier quoted context omitted.

Hi @the_duke, disclaimer I am one of the authors of an open-source solution ( https://github.com/polyaxon/polyaxon ) that specializes in the experimentation and automation phase of the data-science lifecycle. Our tool provides exactly the kind of abstraction you mentioned: * Training, data operations, and interactive workspaces ( https://polyaxon.com/docs/experimentation/ ) * A scalable history and comparison table (…

> but several companies use Polyaxon with other tools like Kubeflow or it can be used with MetaFlow for the pipelines part. Isn't that what the parent you are replying to is talking about with "Right now you have to cobble this together from different tools that are all pretty suboptimal."

Our tool provides several solutions, however, we do not force users to use all of these abstractions. It's very important for us that our product is interoperable with the rest of the ecosystem.

If a company is already using a pipelining tool, a visualization tool, or a data management tool, Polyaxon will work and integrate with those tools seamlessly.

That being said, and I fully understand where the OP is coming from, there are several companies not interested in managing several solutions and all the complexity that comes with the infrastructure, deployment, maintenance, upgrades, user facing clients, authn/authz, permissions... Polyaxon provides the right abstractions for covering the experimentation and the automation phase.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#98

Post by jerzyt »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

Netflix's biggest problem is not the recommendation engine, but their lack of content. If they allowed you to filter out the junk you don't want to see and what you've already seen, there would be hardly anything left. They've been losing rights to stream left and right. That's why there's such a rush to produce their own content in other countries. Some of that is halfway decent, but there's a lot of formulaic, repetitive stuff.

And while saying that, I appreciate their high level technical staff. These decisions are made by bean counters.

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#99

Post by jlkim »

vtuulos wrote: post #5

If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

Are there any built-in dashboards for such actions as querying or analysing model versions and the metadata around them?

Re: Netflix's Metaflow: Reproducible machine learning pipelines

#100

Post by weego »

toxik wrote: post #2

Edit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with ba…

I don't think it's the worst, it's just more that you've made the assumption that Netflix wants a pure set of recommendations. Marketing and business drivers will always trump algo results, so what you're seeing are mostly artificial boosts given to globally 'hot' properties.
Post reply on HN