Maestro: Netflix's Workflow Orchestrator

This is a really great-looking project. I know I've considered building (a probably worse) version of exactly this on almost every mixed ML + Data Engineering project I've ever worked on.

I'm looking forward to testing it out.

Re: Maestro: Netflix's Workflow Orchestrator

#5

Post by oneplane » Mon, Jul 22, 2024, 7:49 PM UTC

Looks a bit like Argo Workflows combined with Argo Events. Makes sense to have so many projects and products converge around the same endstate.

Re: Maestro: Netflix's Workflow Orchestrator

#6

Post by indiv0 » Mon, Jul 22, 2024, 8:04 PM UTC

Is this meaningfully different from Conductor (which they archived a while back)? Browsing through the code I see quite a few similarities. Plus the use of JSON as the workflow definition language.

Re: Maestro: Netflix's Workflow Orchestrator

#7

Post by Sparkyte » Mon, Jul 22, 2024, 8:05 PM UTC

Whats the difference of this and enqueue work into a queue then waiting for a job to pick it up at a scheduled time? Not saying build a Kafka cluster to serve this but most cloud providers have queuing tools.

Re: Maestro: Netflix's Workflow Orchestrator

#8

Post by dboreham » Mon, Jul 22, 2024, 8:05 PM UTC

Interesting. My team recently built a thing for managing long running, multi-machine, restartable, cascading batch jobs in an unrelated vehicle. Had no idea it was a category.

Re: Maestro: Netflix's Workflow Orchestrator

#9

Post by meliora245 » Mon, Jul 22, 2024, 8:06 PM UTC

why would one consider this over something more established such as Temporal, also I see Maestro is written in Java vs Temporal's Go

Re: Maestro: Netflix's Workflow Orchestrator

#10

Post by makestuff » Mon, Jul 22, 2024, 8:24 PM UTC

iamsanteri wrote: post #3

So will this serve as a stand-in replacement for something like Airflow?

Yeah, also curious if this is meant as a replacement for Airflow.