Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

141–150 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#141

Post by archerx »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

This is a naive view, other people’s code is even more of a liability. Look at crowdstrike and opensource infiltrations. Using opensource software doesn’t magically grant you security nor stability.

Re: Maestro: Netflix's Workflow Orchestrator

#142

Post by jcgrillo »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

> People need to realize that code is a liability

Code that you own and intimately understand is less of a liability than some 3rd party dependency (paid or free). Stitching together a patchwork of dependencies is not likely the optimal result. The more aligned your codebase is with the problem you're trying to solve the better, and if functionality is core to your business better to own than borrow or rent.

Re: Maestro: Netflix's Workflow Orchestrator

#143

Post by mikepurvis »

Fidelix wrote: post #127
gregoriol wrote: post #122

Earlier quoted context omitted.

Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution, they just publish their code and continue the project as their tool. They will not have incentive to add features that are not useful to their business even if it useful to the community (if provided by a PR for example), because all the developers of the project are employed by the same company and th…

It has the Apache license, definitely Open Source. https://github.com/Netflix/maestro/blob/main/LICENSE

I think the contention here is more about whether it's an open project— does it have an open bugtracker, an open project management structure, clear governance, etc.

It not having those things is fine, and eventually someone may still take the source and create an open project around it. But understanding that is a Netflix project helps calibrate people's understanding around whether the model when you find a bug is going to be "fork, fix, and run the fork indefinitely" or "fork, fix, contribution accepted, drop fork and return to upstream."

Re: Maestro: Netflix's Workflow Orchestrator

#144

Post by ensignavenger »

rubenfiszel wrote: post #136

Founder of https://windmill.dev here which share many similarities with Maestro. > Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. It oversees the entire lifecycle of a workflow, from start to finish, including retries, queuing, task distribution to compute engines, etc.. Users can…

Thanks for the great comparison! While Meastro is Apache licensed, if it depends on CockroachDB, Cokroach itslef isn't even Open Source, so that isn't great. I would rather have an AGPL codebase than a non open source dependency. Of course overtime some one could add alternative DB support.

Re: Maestro: Netflix's Workflow Orchestrator

#145

Post by fragmede »

bhawks wrote: post #100
patmorgan23 wrote: post #98

Earlier quoted context omitted.

A truck also comes with a maintenance liability if you want to continue getting value out of it, just like code.

Liabilities are obligations of a company to pay money owed to a lender as a result of a previous transaction. You are describing an operating expense which has an entirely different nature than a liability. 'comes with a maintenance liability' is a handwaving statement that means practically nothing without a ton of contextual information. A true liability has a contractual set of obligations to pay defined amounts o…

> No one is going to come after you for not changing the oil on your truck

Several parties will come after you for not changing the oil on your semi-truck that is being used professionally for freight, starting with your driver, your insurance company, and the US Department of Transportation (DOT), specifically the Federal Motor Carrier Safety Administration (FMCSA), with whom you have m have to provide maintenance records. Trucking is a highly regulated industry, and after Crowdstrike, software engineering is only going to get more regulated, not less.

Re: Maestro: Netflix's Workflow Orchestrator

#146

Post by jamra »

ensignavenger wrote: post #144
rubenfiszel wrote: post #136

Founder of https://windmill.dev here which share many similarities with Maestro. > Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. It oversees the entire lifecycle of a workflow, from start to finish, including retries, queuing, task distribution to compute engines, etc.. Users can…

Thanks for the great comparison! While Meastro is Apache licensed, if it depends on CockroachDB, Cokroach itslef isn't even Open Source, so that isn't great. I would rather have an AGPL codebase than a non open source dependency. Of course overtime some one could add alternative DB support.

I really wonder why they didn’t choose something like RocksDB for more speed.

Re: Maestro: Netflix's Workflow Orchestrator

#149

Post by MetaWhirledPeas »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

> code is a liability

3rd parties are also a liability. Pick your poison. Trust in unknown individuals, trust in megacorps, or trust your own people. Choosing wisely is why people get paid the big bucks.

Re: Maestro: Netflix's Workflow Orchestrator

#150

Post by swyx »

alfalfasprout wrote: post #55
hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

The issue is that "workflow orchestration" is a broad problem space. Companies need to address a lot of disparate issues and so any solution ends up being a giant product w/ a lot of associated functionality and heavily opinionated as it grows into a big monolith. This is why almost universally folks are never happy. In reality there are five main concerns: 1. Resource scheduling-- "I have a job or collection of jobs…

great insight, appreciate this. would also point out logging/event sourcing for "free" observability
Post reply on HN