Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

101–110 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#101

Post by cortesoft »

slt2021 wrote: post #74
cortesoft wrote: post #66

Earlier quoted context omitted.

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source t…

So Netflix expects open source community to pick up the maintenance tab ? I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix: 1. They claim it is distributed system, but it is just a regular Java crud with SQL backend 2. Java-like DSL with parser and classloader (why…

> So Netflix expects open source community to pick up the maintenance tab?

Isn’t this the deal with all open source? They are giving something (the code and access to the project) in return for help maintaining it?

No one is being forced to do anything. It is not like there is some open source contributor somewhere now saying, “oh damn, now I have to maintain this, too?”

If people like it and find value in it, they can help contribute to the project in ways they want. Netflix gets to use those contributions, in return for letting people use their contributions. That is just how open source works.

Re: Maestro: Netflix's Workflow Orchestrator

#102

Post by pm90 »

fragmede wrote: post #95
pm90 wrote: post #57

Earlier quoted context omitted.

It’s likely because we haven’t yet found a workflow engine/orchestrator thats capable of handling diverse tasks while still being easy to understand and operate. It’s really easy to build a custom workflow engine and optimize it for specific use cases. I think we haven’t yet seen a convergence simply because this tool hasn’t yet been built. Consider the recent rise of tools that quickly dominated their fields: Terraf…

slurm? airflow?

airflow is notoriously hard to operate https://news.ycombinator.com/item?id=31482217

Re: Maestro: Netflix's Workflow Orchestrator

#103

Post by bhawks »

cbsmith wrote: post #80
bhawks wrote: post #72

Earlier quoted context omitted.

Trucks are literally an asset - you can't do depreciation on a liability. The only way a 'truck' could be a liability is a lease for said truck. There are plenty of economically rationale reasons why a company may own more trucks they strictly need to manage delivery. For example wanting to handle seasonal bursts, wanting to ensure reliability, preparing for an expansion, being able to lease capacity to other busines…

> Please stop misusing accounting concepts. Assets can also be liabilities. The mortgages in a mortgage backed security is both an asset and a liability, as was only too well demonstrated in 2008... It's an asset in the security portfolio, but until you sell the security, it's a liability for whomever is securitizing it.

In the GFC the government literally created the Troubled _Asset_ Relief Program. Those MBSs were assets and didn't magically become liabilities.

The problem was the market value of those assets plummeted because no one expected them to generate the agreed upon cash flows because the underlying loans were going into correlated defaults. Despite all this the only party that saw the mortgage as a liability is the individual who's responsibility it was to make a monthly payment on said mortgage.

Outside of swaps and other derivatives financial instruments and other properties don't magically switch from being an asset to being a liability based on random external factors.

This conversation is like accountants talking about processes, threads, fibers and context switching... very imprecisely.

Re: Maestro: Netflix's Workflow Orchestrator

#104

Post by slt2021 »

narism wrote: post #93
slt2021 wrote: post #87

Earlier quoted context omitted.

Amazon actually steals a lot of open source and repackages it as a “managed AWS service”, they literally deployed managed Airflow as soon as it became popular. The whole aws reinvent is repackaged whatever open source project is trending, hiding control plane from the user and instead expose it via AWS control plane and charge people per usage instead of per server

Can you steal something that is given away freely? People are buying those services so they must be providing some value.

I was just replying to the comment above that amazon somehow rolls their own stuff and gives back to the community by open sourcing their systems.

Amazon’s approach is the opposite: steal open source repo and make $$$ off of open source contributors’ labor

Re: Maestro: Netflix's Workflow Orchestrator

#105

Post by bhawks »

narism wrote: post #93
slt2021 wrote: post #87

Earlier quoted context omitted.

Amazon actually steals a lot of open source and repackages it as a “managed AWS service”, they literally deployed managed Airflow as soon as it became popular. The whole aws reinvent is repackaged whatever open source project is trending, hiding control plane from the user and instead expose it via AWS control plane and charge people per usage instead of per server

Can you steal something that is given away freely? People are buying those services so they must be providing some value.

I can't find a single line in the Apache software license that would indicate that Amazon is breaking any agreements set forth for Airflow.

Re: Maestro: Netflix's Workflow Orchestrator

#106

Post by slt2021 »

bushbaba wrote: post #99
slt2021 wrote: post #87

Earlier quoted context omitted.

Amazon actually steals a lot of open source and repackages it as a “managed AWS service”, they literally deployed managed Airflow as soon as it became popular. The whole aws reinvent is repackaged whatever open source project is trending, hiding control plane from the user and instead expose it via AWS control plane and charge people per usage instead of per server

Accept they also provide the security, billing/invoicing, IaaC, support, provisioning, scaling, list goes on. As for pay per server vs pay per usage. Heck you know Amazon actually bills the team who caused the cost. And gives finance a report on how much each team is spending and on what. Good luck doing that on prem.

The question is how much do they give back to the open source community, after making boatloads of $$$ off of opensource contributions and whether their model is sustainable and healthy for the FOSS movement

Re: Maestro: Netflix's Workflow Orchestrator

#107

Post by SOLAR_FIELDS »

alfalfasprout wrote: post #55
hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

The issue is that "workflow orchestration" is a broad problem space. Companies need to address a lot of disparate issues and so any solution ends up being a giant product w/ a lot of associated functionality and heavily opinionated as it grows into a big monolith. This is why almost universally folks are never happy. In reality there are five main concerns: 1. Resource scheduling-- "I have a job or collection of jobs…

The issue indeed is that "workflow orchestration" is a broad problem space. I would argue that the solution is not this:

> I don't think that you can just spin up a startup to deliver this as a "solution". This needs to be solved with an open source ecosystem of good pluggable modular components.

But rather more specialized tools that solve specific issues.

What you describe just sounds like a better implemented version of Airflow or the over 100 other systems that are actively trying to be this today (Flyte, Dagster, Prefect, Argo Workflows, Kubeflow, Nifi, Oozie, Conductor, Cadence, Temporal, Step Functions, Logic Apps, your CI system of choice has their own, need I continue, that is not even scratching the surface). Most of those have some sort of "plugin" ecosystem for custom code, in varying degrees of robustness.

For what it is worth, everyone and their mom thinks they can make and wants to be this orchestrator. It's a problem that is just so generic and such a wide net that you end up with annoying-to-use building blocks because everyone wants to architecture astronaut themselves into being the generic workflow orchestration engine. The ultimate system design trap: Something so fundamentally easy to grok and conceptualize that you can PoC one in hours or days, but near infinite possibilities of what you can do with it, resulting in near infinite edge cases.

Instead, I'd rather companies just focus on the problem space that it lends itself to. Instead of Dagster saying "Automate any workflow" and try to capture that space, just make building blocks for data engineering workflows and get really good at that. Instead of Github Actions being a generic "workflow engine" just have it really good at making CI workflow building blocks.

But we can't have it that way. Because then some architecture astronaut will come around and design a generic workflow engine for orchestrating your domain specific workflow engines and say that you no longer need those.

Actually I think I just convinced myself that what you are suggesting actually IS the right way. If companies just said "we will provide an Airflow plugin" instead of building their own damn Airflow this would be easy. But we won't ever have that either. What we really need is some standards around that. Like if CNCF got together and got tired of this and said "This is THE canonical and supported engine for Kube workflows, bring your plugins here if you want us to pump you up". That might work. They've usually had better luck with putting people in lockstep in the Kube ecosystem at least than Apache has historically for more general FOSS stuff. Probably because the problem space there is more limited.

Re: Maestro: Netflix's Workflow Orchestrator

#108

Post by antishatter »

Anyone have a recommendation for a workflow orchestrator for single server deployments? Looking at running a project at home and for certain pieces think it would be easiest to orchestrate with a tool like Maestro or Airflow but they’re basically set up to run in clusters with admins to manage them.

Re: Maestro: Netflix's Workflow Orchestrator

#109

Post by torrance »

andbberger wrote: post #63

slightly off topic, but there is dire need for a scientific "workflow manager" built to FAANG engineering standards attuned for the needs of academia (ie primarily designed to facilitate execution of DAGs on clusters). The airflows of the world have complex unnecessary features and require extensive kitbashing to plug into slurm and the academic side of things is a huge mess. Snakemake comes the closest but suffers f…

What about Nextflow?

Re: Maestro: Netflix's Workflow Orchestrator

#110

Post by darkwater »

rjh29 wrote: post #75
cortesoft wrote: post #66

Earlier quoted context omitted.

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source t…

Usually the corporate needs differ too much and they end up keeping their own fork anyway. Netflix has the resources to maintain this. It's probably more a PR move for their hiring division.

Absolutely this. It literally happened to me with a Netflix OSS we were using at work. I found a bug that was biting us, opened a ticket with a PR attached with a possible fix and got an answer after a few months "ah yeah we fixed this in our internal version time ago, thanks, will merge it now".
Post reply on HN