Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

151–160 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#151

Post by swyx »

trustno2 wrote: post #125
meliora245 wrote: post #9

why would one consider this over something more established such as Temporal, also I see Maestro is written in Java vs Temporal's Go

Temporal's go is... something. They used to use Java (I think), then they switched to Go, and the Go is very Java-like. Or maybe I just don't know Fx. https://github.com/temporalio/temporal/blob/main/service/mat... The issue we hit with Temporal - again and again - is that it's very under-documented, and it's something you install at the core of your business, yet it's really hard to understand what is going on, thro…

no just the SDK is Java. temporal is 99% Golang, even at Uber https://github.com/uber/cadence

Re: Maestro: Netflix's Workflow Orchestrator

#152

Post by swyx »

troebr wrote: post #18
jekude wrote: post #15

Seems like they re-engineered Temporal: https://temporal.io/

They did use Temporal at Netflix, they gave a couple presentations 2 years ago. I think this is very much not-Temporal because it relies on a DSL instead of workflow as code. I don't know if it's a scale-thing, I'm not a workflow expert but this seems more in line with the map-reduce of yore, as in you get some big fat steps and you coordinate them, although you could have coarse-grained activities in Temporal workfl…

former temporal employee here. netflix is very big, temporal-at-netflix always coexisted with other orchestration solutions including conductor

> I think this is very much not-Temporal because it relies on a DSL instead of workflow as code.

yup you get it. maestro defines things as json, which just inherently limits how you can write and test it with your normal app code

Re: Maestro: Netflix's Workflow Orchestrator

#153

Post by paxys »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

> I would rather use off-the-shelf open source stuff with long history of maintenance and improvement

Where do you think this "off-the-shelf open source stuff" comes from exactly?

Re: Maestro: Netflix's Workflow Orchestrator

#154

Post by mwfussell »

digger495 wrote: post #135
slt2021 wrote: post #74

Earlier quoted context omitted.

So Netflix expects open source community to pick up the maintenance tab ? I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix: 1. They claim it is distributed system, but it is just a regular Java crud with SQL backend 2. Java-like DSL with parser and classloader (why…

People want an alternative to things like Temporal, and don't want to handle DAGs with Kafka Streams.

An alternative for a code based workflow like Temporal are Dapr workflows (https://docs.dapr.io/developing-applications/building-blocks...), where you write a set of code based activities into a graph and these can have fan-in, fan-out, sequential patterns etc. Its all code in several supported languages and because Dapr also has building block APIs for pub/sub, service invocation, secrets as well as connecting to underlying infrastructure you can combine workflow with these APIs to build an overall solution.

Re: Maestro: Netflix's Workflow Orchestrator

#155

Post by b800h »

gregoriol wrote: post #122
rjh29 wrote: post #75

Earlier quoted context omitted.

Usually the corporate needs differ too much and they end up keeping their own fork anyway. Netflix has the resources to maintain this. It's probably more a PR move for their hiring division.

Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution, they just publish their code and continue the project as their tool. They will not have incentive to add features that are not useful to their business even if it useful to the community (if provided by a PR for example), because all the developers of the project are employed by the same company and th…

It's absolutely open source, under the Apache license. If a nascent community takes it and runs with it then the public version will become dominant.

Re: Maestro: Netflix's Workflow Orchestrator

#156

Post by pandaswo »

mwfussell wrote: post #154
digger495 wrote: post #135

Earlier quoted context omitted.

People want an alternative to things like Temporal, and don't want to handle DAGs with Kafka Streams.

An alternative for a code based workflow like Temporal are Dapr workflows ( https://docs.dapr.io/developing-applications/building-blocks... ), where you write a set of code based activities into a graph and these can have fan-in, fan-out, sequential patterns etc. Its all code in several supported languages and because Dapr also has building block APIs for pub/sub, service invocation, secrets as well as connecting to…

How scalable is this one? Hated airflow for the scalability issues.

Re: Maestro: Netflix's Workflow Orchestrator

#157

Post by arresin »

swyx wrote: post #151
trustno2 wrote: post #125

Earlier quoted context omitted.

Temporal's go is... something. They used to use Java (I think), then they switched to Go, and the Go is very Java-like. Or maybe I just don't know Fx. https://github.com/temporalio/temporal/blob/main/service/mat... The issue we hit with Temporal - again and again - is that it's very under-documented, and it's something you install at the core of your business, yet it's really hard to understand what is going on, thro…

no just the SDK is Java. temporal is 99% Golang, even at Uber https://github.com/uber/cadence

They're saying the code style is java like and I agree. Whether or not that's a bad thing depends on your opinion of java and java coding styles.

Re: Maestro: Netflix's Workflow Orchestrator

#158

Post by arresin »

rubenfiszel wrote: post #136

Founder of https://windmill.dev here which share many similarities with Maestro. > Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. It oversees the entire lifecycle of a workflow, from start to finish, including retries, queuing, task distribution to compute engines, etc.. Users can…

Anyone considering windmill needs to look at this first:

https://www.windmill.dev/docs/advanced/local_development.

Why do I need to "sync" with windmill? Why is there an IDE built into windmill? Why is this so convoluted? It's like it's starting with the goal of lock-in before even developing a good product or finding market fit.

Re: Maestro: Netflix's Workflow Orchestrator

#159

Post by reinhardt »

turtle4 wrote: post #124
artwr wrote: post #114

Earlier quoted context omitted.

Agreed. To be fair, I doubt Maestro will take off like Airflow did. Airflow filled a void of an easier orchestrator for Big Data with a prettier UI than the competitors of the time (Oozie, Luigi), implementing some UX patterns which had been tested at scale at Facebook with data swarm. The field is quite a bit more crowded now.

Seems like you have some experience with the orchestrator offerings. Airflow still the way to go, or would you recommend something else for someone just starting down the path of selecting and implementing a data orchestrator?

I haven't used Airflow for years but it used to be quite clunky, not sure how much it's improved since. I'd look into Prefect and/or Dagster first, both are more modern alternatives built with Airflow's shortcomings in mind.
Post reply on HN