Dagster is a better alternative, because of its asset first philosophy. Task based workflows are still available if you really need it.
Maestro: Netflix's Workflow Orchestrator
121–130 of 165 posts
Re: Maestro: Netflix's Workflow Orchestrator
#122Earlier quoted context omitted.
Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source t…
Usually the corporate needs differ too much and they end up keeping their own fork anyway. Netflix has the resources to maintain this. It's probably more a PR move for their hiring division.
Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution, they just publish their code and continue the project as their tool. They will not have incentive to add features that are not useful to their business even if it useful to the community (if provided by a PR for example), because all the developers of the project are employed by the same company and this company doesn't have any reason to review and fix code that is not part of their business.
Re: Maestro: Netflix's Workflow Orchestrator
#123Earlier quoted context omitted.
Usually the corporate needs differ too much and they end up keeping their own fork anyway. Netflix has the resources to maintain this. It's probably more a PR move for their hiring division.
Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution, they just publish their code and continue the project as their tool. They will not have incentive to add features that are not useful to their business even if it useful to the community (if provided by a PR for example), because all the developers of the project are employed by the same company and th…
> Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution
It's open source, and they don't have to accept external contributions. Terms have a well-defined meaning, please refrain from calling open source code not open source, and not open source code, open source.
Re: Maestro: Netflix's Workflow Orchestrator
#124Earlier quoted context omitted.
But when Airbnb created airflow, you could have said the same. It’s just later in its lifecycle.
Agreed. To be fair, I doubt Maestro will take off like Airflow did. Airflow filled a void of an easier orchestrator for Big Data with a prettier UI than the competitors of the time (Oozie, Luigi), implementing some UX patterns which had been tested at scale at Facebook with data swarm. The field is quite a bit more crowded now.
Seems like you have some experience with the orchestrator offerings. Airflow still the way to go, or would you recommend something else for someone just starting down the path of selecting and implementing a data orchestrator?
Re: Maestro: Netflix's Workflow Orchestrator
#125why would one consider this over something more established such as Temporal, also I see Maestro is written in Java vs Temporal's Go
Temporal's go is... something. They used to use Java (I think), then they switched to Go, and the Go is very Java-like.
Or maybe I just don't know Fx.
https://github.com/temporalio/temporal/blob/main/service/mat...
The issue we hit with Temporal - again and again - is that it's very under-documented, and it's something you install at the core of your business, yet it's really hard to understand what is going on, through all the layers and through the very obtuse documentation.
Maestro has... no documentation? OK Temporal wins by default.
Re: Maestro: Netflix's Workflow Orchestrator
#126So will this serve as a stand-in replacement for something like Airflow?
I'm also missing comparisons to other existing tools like airflow, dagster, mlflow...
Re: Maestro: Netflix's Workflow Orchestrator
#127Earlier quoted context omitted.
Usually the corporate needs differ too much and they end up keeping their own fork anyway. Netflix has the resources to maintain this. It's probably more a PR move for their hiring division.
Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution, they just publish their code and continue the project as their tool. They will not have incentive to add features that are not useful to their business even if it useful to the community (if provided by a PR for example), because all the developers of the project are employed by the same company and th…
It has the Apache license, definitely Open Source. https://github.com/Netflix/maestro/blob/main/LICENSE
Re: Maestro: Netflix's Workflow Orchestrator
#128I'm building something in the space (orchestra) so here's my take:
Folks making stuff open source and building in the open is obviously brilliant, but when it comes to "orchestrators" (as this is, and identifies) there is already so much that has been before (Airflow and so on) it's quite hard to see how this actually adds anything to the space other than another option nobody is ever going to use in a commercial setting.
Shameless plug: https://getorchestra.io
Re: Maestro: Netflix's Workflow Orchestrator
#129Earlier quoted context omitted.
Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution, they just publish their code and continue the project as their tool. They will not have incentive to add features that are not useful to their business even if it useful to the community (if provided by a PR for example), because all the developers of the project are employed by the same company and th…
> Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution It's open source, and they don't have to accept external contributions. Terms have a well-defined meaning, please refrain from calling open source code not open source, and not open source code, open source.
You are arguing the difference between the letter and spirit of the law.
Re: Maestro: Netflix's Workflow Orchestrator
#130Anyone have a recommendation for a workflow orchestrator for single server deployments? Looking at running a project at home and for certain pieces think it would be easiest to orchestrate with a tool like Maestro or Airflow but they’re basically set up to run in clusters with admins to manage them.
Windmill is pretty lightweight and easy to deploy. https://www.windmill.dev/ you can configure it to have a single worker on the same server as the ui and database.