Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

131–140 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#131

Post by nicce »

cortesoft wrote: post #101
slt2021 wrote: post #74

Earlier quoted context omitted.

So Netflix expects open source community to pick up the maintenance tab ? I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix: 1. They claim it is distributed system, but it is just a regular Java crud with SQL backend 2. Java-like DSL with parser and classloader (why…

> So Netflix expects open source community to pick up the maintenance tab? Isn’t this the deal with all open source? They are giving something (the code and access to the project) in return for help maintaining it? No one is being forced to do anything. It is not like there is some open source contributor somewhere now saying, “oh damn, now I have to maintain this, too?” If people like it and find value in it, they c…

> Isn’t this the deal with all open source?

If Netflix still heavily uses this internally, they should still do the most maintenance. Others contribute based on their own needs.

Re: Maestro: Netflix's Workflow Orchestrator

#132

Post by _Wintermute »

torrance wrote: post #109
andbberger wrote: post #63

slightly off topic, but there is dire need for a scientific "workflow manager" built to FAANG engineering standards attuned for the needs of academia (ie primarily designed to facilitate execution of DAGs on clusters). The airflows of the world have complex unnecessary features and require extensive kitbashing to plug into slurm and the academic side of things is a huge mess. Snakemake comes the closest but suffers f…

What about Nextflow?

Nobody wants to write or debug groovy, especially scientists who are used to python. It also causes havoc on a busy SLURM scheduler with its lack of array jobs (heard this is being fixed soon).

Re: Maestro: Netflix's Workflow Orchestrator

#133

Post by otabdeveloper4 »

hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

> why engineers are so keen on building their own workflow engines

Because all the existing ones suck.

(We built our own tiny one two. We need tight integration with systemd jobs and cgroups, and existing solutions don't do that.)

Re: Maestro: Netflix's Workflow Orchestrator

#134

Post by jjuliano »

slt2021 wrote: post #74
cortesoft wrote: post #66

Earlier quoted context omitted.

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source t…

So Netflix expects open source community to pick up the maintenance tab ? I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix: 1. They claim it is distributed system, but it is just a regular Java crud with SQL backend 2. Java-like DSL with parser and classloader (why…

> So Netflix expects open source community to pick up the maintenance tab ?

I think the notion of open sourcing a project, is you are literally asking at the community for help and that the community will naturally help you with the maintenance.

Re: Maestro: Netflix's Workflow Orchestrator

#135

Post by digger495 »

slt2021 wrote: post #74
cortesoft wrote: post #66

Earlier quoted context omitted.

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source t…

So Netflix expects open source community to pick up the maintenance tab ? I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix: 1. They claim it is distributed system, but it is just a regular Java crud with SQL backend 2. Java-like DSL with parser and classloader (why…

People want an alternative to things like Temporal, and don't want to handle DAGs with Kafka Streams.

Re: Maestro: Netflix's Workflow Orchestrator

#136

Post by rubenfiszel »

Founder of https://windmill.dev here which share many similarities with Maestro.

> Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. It oversees the entire lifecycle of a workflow, from start to finish, including retries, queuing, task distribution to compute engines, etc.. Users can package their business logic in various formats such as Docker images, notebooks, bash script, SQL, Python, and more. Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc.

You could replace Maestro with Windmill here and it would be precisely correct. Their rollup is what we call the openflow state.

Main differences I see:

- Windmill is written in Rust instead of Java.

- Maestro relies on CockroachDB for state and us Postgresql for everything (state but also queue). I can see why they would use CockroachDB, we had to rollout our own sharding algorithms to make Windmill horizontally scale on our very large scale customer instances

- Maestro is Apache 2.0 vs Windmill AGPL which is less friendly

- It's backed by Netflix so infinite money but although we are profitable, we are a much smaller company

- Maestro doesn't have extensive docs about self-hosting on k8s or docker-compose and either there is no UI to build stuff, or the UI is not yet well surfaced in their documentation

But overall, pretty cool stuff to open-source, will keep an eye on it and benchmark it asap

Re: Maestro: Netflix's Workflow Orchestrator

#137

Post by rwky »

rubenfiszel wrote: post #136

Founder of https://windmill.dev here which share many similarities with Maestro. > Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. It oversees the entire lifecycle of a workflow, from start to finish, including retries, queuing, task distribution to compute engines, etc.. Users can…

Been using windmill for a few months and so far it's rock solid keep it up!

Re: Maestro: Netflix's Workflow Orchestrator

#138

Post by delecti »

bhawks wrote: post #72
delecti wrote: post #60

Earlier quoted context omitted.

Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.

Trucks are literally an asset - you can't do depreciation on a liability. The only way a 'truck' could be a liability is a lease for said truck. There are plenty of economically rationale reasons why a company may own more trucks they strictly need to manage delivery. For example wanting to handle seasonal bursts, wanting to ensure reliability, preparing for an expansion, being able to lease capacity to other busines…

I'm not using liability in the accounting context, but in the colloquial one.

> a person or thing whose presence or behavior is likely to cause embarrassment or put one at a disadvantage.

Code is absolutely a liability. Code deteriorates as conditions change, and unchanged code also becomes more vulnerable in a way that conventional objects can't.

Re: Maestro: Netflix's Workflow Orchestrator

#139

Post by [deleted] »

sramam wrote: post #129
mhitza wrote: post #123

Earlier quoted context omitted.

> Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution It's open source, and they don't have to accept external contributions. Terms have a well-defined meaning, please refrain from calling open source code not open source, and not open source code, open source.

You are arguing the difference between the letter and spirit of the law.

[deleted]

Re: Maestro: Netflix's Workflow Orchestrator

#140

Post by homarp »

sramam wrote: post #129
mhitza wrote: post #123

Earlier quoted context omitted.

> Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution It's open source, and they don't have to accept external contributions. Terms have a well-defined meaning, please refrain from calling open source code not open source, and not open source code, open source.

You are arguing the difference between the letter and spirit of the law.

you can fork it and continue dev in case they change their mind, it is open source.

any other 'source available' licenses would not (legally) let you do that.

Post reply on HN