Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

111–120 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#111

Post by bhawks »

slt2021 wrote: post #71
bhawks wrote: post #54

Earlier quoted context omitted.

> with long history of maintenance and improvement, That is a huge load bearing statement. Do you plan on any contributions back to the community yourself? Build vs. buy is always an important conversation but claiming that the 'buy'-side path has perfectly 0 maintenance and reliability costs reeks of naivety.

If I needed container orchestration I would use k8s. I can improve it and even propose patches/bugs or chip into opensource maintainers fund. I wont write my own orchestrator, especialy being in a streaming business. Thats what I meant, doesnt even necessarily Build-vs-Buy, but rather Use-Open-Source-and-Contribute or Reinvent-the-wheel-for-L6-promo-and-then-opensource ?? Would the world be better with 10 workflow or…

Netflix is building a workflow orchestrator not a container orchestrator. The viable alternative would be Airflow or maybe something like Temporal. K8s alone isn't going to meet the need in this case.

Does the world need another workflow orchestrator? Who knows - some folks at Netflix seem willing to pay a handful of engineers $ to do so. Good luck to them

Re: Maestro: Netflix's Workflow Orchestrator

#112

Post by beanjuiceII »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

Off the shelf come with its own set of burdens, not always sunshine rainbows and loli's

Re: Maestro: Netflix's Workflow Orchestrator

#113

Post by wodenokoto »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

Aren’t most of those things developed in house at tech giants and later open sourced?

Re: Maestro: Netflix's Workflow Orchestrator

#114

Post by artwr »

hulahoof wrote: post #97
slt2021 wrote: post #77

Earlier quoted context omitted.

airflow and Temporal has teams dedicated to maintain and extend their system. And these systems are business critical for astronomer/temporal, respectively. And they develop them in a way that works for many customers and use cases, not just netflix. But for netflix this is just another auxillary system, out of many others. Just a nice GUI to schedule cron jobs basically, does it make sense to sink resources into cus…

But when Airbnb created airflow, you could have said the same. It’s just later in its lifecycle.

Agreed.

To be fair, I doubt Maestro will take off like Airflow did.

Airflow filled a void of an easier orchestrator for Big Data with a prettier UI than the competitors of the time (Oozie, Luigi), implementing some UX patterns which had been tested at scale at Facebook with data swarm.

The field is quite a bit more crowded now.

Re: Maestro: Netflix's Workflow Orchestrator

#115

Post by andbberger »

torrance wrote: post #109
andbberger wrote: post #63

slightly off topic, but there is dire need for a scientific "workflow manager" built to FAANG engineering standards attuned for the needs of academia (ie primarily designed to facilitate execution of DAGs on clusters). The airflows of the world have complex unnecessary features and require extensive kitbashing to plug into slurm and the academic side of things is a huge mess. Snakemake comes the closest but suffers f…

What about Nextflow?

I considered Nextflow before begrudgingly settling on snakemake for my current project. Didn't record why... possibly because snakemake was already a known quantity and I was under time pressure or because I felt the task DAG would be difficult to specify in WDL. It's certainly the most mature of the bunch.

Re: Maestro: Netflix's Workflow Orchestrator

#116

Post by andbberger »

slt2021 wrote: post #89
andbberger wrote: post #63

slightly off topic, but there is dire need for a scientific "workflow manager" built to FAANG engineering standards attuned for the needs of academia (ie primarily designed to facilitate execution of DAGs on clusters). The airflows of the world have complex unnecessary features and require extensive kitbashing to plug into slurm and the academic side of things is a huge mess. Snakemake comes the closest but suffers f…

Academia better to learn k8s and one of the k8s-native workflow orchestrators. This is as close to FAANG grade and open source as they can get, and arguably a bit better than this repo

for better or worse slurm is the status quo for HPC. it works, every university has a slurm cluster, people already know how to use it

Re: Maestro: Netflix's Workflow Orchestrator

#118

Post by ssfak »

antishatter wrote: post #108

Anyone have a recommendation for a workflow orchestrator for single server deployments? Looking at running a project at home and for certain pieces think it would be easiest to orchestrate with a tool like Maestro or Airflow but they’re basically set up to run in clusters with admins to manage them.

For Python tasks you can check Prefect, among others..

Re: Maestro: Netflix's Workflow Orchestrator

#119

Post by xcdzvyn »

slt2021 wrote: post #74
cortesoft wrote: post #66

Earlier quoted context omitted.

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source t…

So Netflix expects open source community to pick up the maintenance tab ? I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix: 1. They claim it is distributed system, but it is just a regular Java crud with SQL backend 2. Java-like DSL with parser and classloader (why…

[dead]

Re: Maestro: Netflix's Workflow Orchestrator

#120

Post by katrotz »

antishatter wrote: post #108

Anyone have a recommendation for a workflow orchestrator for single server deployments? Looking at running a project at home and for certain pieces think it would be easiest to orchestrate with a tool like Maestro or Airflow but they’re basically set up to run in clusters with admins to manage them.

I'd recommend Kestra[1] since it can be run on a single node

[1] https://kestra.io/

Post reply on HN