Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

51–60 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#51

Post by ATMLOTTOBEER »

thfuran wrote: post #41
ilrwbwrkhv wrote: post #37

Earlier quoted context omitted.

Its because Netflix pretends to be a tech company to get the high market cap. So they hire tons of engineers who have nothing to do but rearchitecture the mess their microservices have created. Then there are others who create observability and test harnesses for all of that. When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bulls…

>When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bullshit When is that, exactly? https://www.statista.com/chart/15692/distribution-of-global-...

“Other” in your diagram is mostly porn

Re: Maestro: Netflix's Workflow Orchestrator

#52

Post by bluepizza »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

> People need to realize that code is a liability

This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun.

I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech.

Code is not a liability. Code is what makes a company move its gears.

Re: Maestro: Netflix's Workflow Orchestrator

#53

Post by alfalfasprout »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

I very much disagree with this take-- and the more I've experienced throughout my career the more I'm sure of it.

Companies spend an IMMENSE amount of time and effort adapting sometimes subpar off the shelf solutions to fit their infra and pay an ongoing tax w/ increasing tech debt trying to support them. Often something bespoke and smaller + more tailored would unlock significantly more productivity if the investment is made consciously.

Any code that is written has both assets and liabilities. But to claim it is a distraction and resource sink is a very, very bad take. Every decision to build something in-house needs to be done thoughtfully and deliberately.

Re: Maestro: Netflix's Workflow Orchestrator

#54

Post by bhawks »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

> with long history of maintenance and improvement,

That is a huge load bearing statement.

Do you plan on any contributions back to the community yourself?

Build vs. buy is always an important conversation but claiming that the 'buy'-side path has perfectly 0 maintenance and reliability costs reeks of naivety.

Re: Maestro: Netflix's Workflow Orchestrator

#55

Post by alfalfasprout »

hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

The issue is that "workflow orchestration" is a broad problem space. Companies need to address a lot of disparate issues and so any solution ends up being a giant product w/ a lot of associated functionality and heavily opinionated as it grows into a big monolith. This is why almost universally folks are never happy.

In reality there are five main concerns: 1. Resource scheduling-- "I have a job or collection of jobs to run... allocate them to the machines I have" 2. Dependency solving-- If my jobs have dependencies on each other, perform the topological sort so I can dispatch things to my resource scheduler 3. API/DSL for creating jobs and workflows. I want to define a DAG... sometimes static, sometimes on the fly. 4. Cron-like functionality. I want to be able to run things on a schedule or ad-hoc. 5. Domain awareness-- If doing ETL I want my DAGs to be data aware... if doing ML/AI workflows then I want to be able to surface info about what I'm actually doing with them

No one solution does all these things cleanly. So companies end up building or hacking around off the shelf stuff to deal with the downsides of existing solutions. Hence it's a perpetual cycle of everyone being unhappy.

I don't think that you can just spin up a startup to deliver this as a "solution". This needs to be solved with an open source ecosystem of good pluggable modular components.

Re: Maestro: Netflix's Workflow Orchestrator

#56

Post by rty32 »

thfuran wrote: post #41
ilrwbwrkhv wrote: post #37

Earlier quoted context omitted.

Its because Netflix pretends to be a tech company to get the high market cap. So they hire tons of engineers who have nothing to do but rearchitecture the mess their microservices have created. Then there are others who create observability and test harnesses for all of that. When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bulls…

>When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bullshit When is that, exactly? https://www.statista.com/chart/15692/distribution-of-global-...

What is the methodology of the report?

Just one of the questions I have regarding this -- China has nearly 1.4 billion people, and barely any of them use any of the services here. Instead, they have their own video platforms. And you tell me that none of those platforms use at least the same amount of traffic of Prime Video? I doubt it.

Re: Maestro: Netflix's Workflow Orchestrator

#57

Post by pm90 »

hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

It’s likely because we haven’t yet found a workflow engine/orchestrator thats capable of handling diverse tasks while still being easy to understand and operate.

It’s really easy to build a custom workflow engine and optimize it for specific use cases. I think we haven’t yet seen a convergence simply because this tool hasn’t yet been built.

Consider the recent rise of tools that quickly dominated their fields: Terraform (IaC), Kubernetes (distributed compute). Both systems are hella complex, but they solve hard problems. Generic workflow engines are complex to understand and difficult to operate and offer a middling experience so many folks don’t even bother.

Re: Maestro: Netflix's Workflow Orchestrator

#58

Post by why-el »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

I am confused by this comment:

> open source stuff with long history of maintenance and improvement

improvement and maintenance is continent on usage, and having been used at Netflix, this project is in a better place to have already faced whatever bug you are worried about (and let's be real, 99% of applications wont ever get the luck to exercise code paths sophisticated enough to find bugs Netflix has not found already).

You might be unnecessarily projecting here. You don't have evidence to support that open sourcing this might have been for any other reason than it is simply good for the community to have.

Re: Maestro: Netflix's Workflow Orchestrator

#59

Post by [deleted] »

bhawks wrote: post #54
slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

> with long history of maintenance and improvement, That is a huge load bearing statement. Do you plan on any contributions back to the community yourself? Build vs. buy is always an important conversation but claiming that the 'buy'-side path has perfectly 0 maintenance and reliability costs reeks of naivety.

[deleted]

Re: Maestro: Netflix's Workflow Orchestrator

#60

Post by delecti »

bluepizza wrote: post #52
slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

> People need to realize that code is a liability This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun. I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech. Cod…

Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.
Post reply on HN