Live data from Hacker News

Netflix has open-sourced its Maestro Workflow Orchestrator

github.com

31–40 of 42 posts

Re: Netflix has open-sourced its Maestro Workflow Orchestrator

#31
post #5

I was just thinking didn't Netflix already have its own workflow engine: https://github.com/Netflix/conductor , but found out it's archived/EOL so this makes more sense now.

Conductor was moved here: https://github.com/conductor-oss/conductor

And it is supported! https://orkes-conductor.slack.com

Re: Netflix has open-sourced its Maestro Workflow Orchestrator

#32
post #13
post #2

This appears to be the high level architecture: https://miro.medium.com/v2/resize:fit:4800/format:webp/0*SDt... . To me seems to overengineered for most of the companies outside there, how many people do you need to manage it?

That chart has real Krazam Microservices vibes to it.

Here's a video of Primeagen, a YouTuber who uses to be a dev at Netflix, watching that video and then telling a very similar story that happened at Netflix: https://youtu.be/s-vJcOfrvi0

Re: Netflix has open-sourced its Maestro Workflow Orchestrator

#33
post #2

This appears to be the high level architecture: https://miro.medium.com/v2/resize:fit:4800/format:webp/0*SDt... . To me seems to overengineered for most of the companies outside there, how many people do you need to manage it?

this looks like someone took their systems design interview too seriously

Re: Netflix has open-sourced its Maestro Workflow Orchestrator

#35

1) Scheduling seems to be primitive? (strategy: sequential) 2) That's seems to be just a DAG executor? No high-level frontend? 3) No executiun context? 4) No typings? 5) No concept of a stream? It would be way too ambitious to call this thing an orchestrator, it seems to be just a primitive distributed DAG executor.

Did you miss the sections on how this allows cyclic graphs and not just DAG?

It's hard to take your comment seriously when this was one of the earliest talking points.

Re: Netflix has open-sourced its Maestro Workflow Orchestrator

#36
post #13

Earlier quoted context omitted.

That chart has real Krazam Microservices vibes to it.

Here's a video of Primeagen, a YouTuber who uses to be a dev at Netflix, watching that video and then telling a very similar story that happened at Netflix: https://youtu.be/s-vJcOfrvi0

Brilliant. Currently dealing with webs of microservices myself. It's truly wild to watch something akin to the cambrian explosion with these things.

Re: Netflix has open-sourced its Maestro Workflow Orchestrator

#37
post #24
post #16

Earlier quoted context omitted.

It doesn't build on windows either. It looks to be missing some dependency. > Did not resolve 'com.github.vmg.protogen:protogen-annotations:1.0.0' which is part of the dependency lock state

I get a similar error

I couldn’t find that package on Maven Central either. Though its source is on GitHub.

Re: Netflix has open-sourced its Maestro Workflow Orchestrator

#38
post #3

I was hoping to try this out, but it doesn't build on my mac :/ Here's their tech blog post about it: https://netflixtechblog.com/orchestrating-data-ml-workflows-...

I've created a Docker image for it and onboarded it into my tool called insta-infra[1]. You should be able to run it via:

    ./run.sh maestro
[1] https://github.com/data-catering/insta-infra

Re: Netflix has open-sourced its Maestro Workflow Orchestrator

#39
post #35

1) Scheduling seems to be primitive? (strategy: sequential) 2) That's seems to be just a DAG executor? No high-level frontend? 3) No executiun context? 4) No typings? 5) No concept of a stream? It would be way too ambitious to call this thing an orchestrator, it seems to be just a primitive distributed DAG executor.

Did you miss the sections on how this allows cyclic graphs and not just DAG? It's hard to take your comment seriously when this was one of the earliest talking points.

> In Maestro, a workflow is a DAG (Directed acyclic graph) of individual units of job definition called Steps.

Re: Netflix has open-sourced its Maestro Workflow Orchestrator

#40
post #35

Earlier quoted context omitted.

Did you miss the sections on how this allows cyclic graphs and not just DAG? It's hard to take your comment seriously when this was one of the earliest talking points.

> In Maestro, a workflow is a DAG (Directed acyclic graph) of individual units of job definition called Steps.

The link was changed at some point, from the original Medium article:

> Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc.

https://netflixtechblog.com/maestro-netflixs-workflow-orches...

Post reply on HN