Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

71–80 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#71

Post by slt2021 »

bhawks wrote: post #54
slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

> with long history of maintenance and improvement, That is a huge load bearing statement. Do you plan on any contributions back to the community yourself? Build vs. buy is always an important conversation but claiming that the 'buy'-side path has perfectly 0 maintenance and reliability costs reeks of naivety.

If I needed container orchestration I would use k8s. I can improve it and even propose patches/bugs or chip into opensource maintainers fund. I wont write my own orchestrator, especialy being in a streaming business.

Thats what I meant, doesnt even necessarily Build-vs-Buy, but rather Use-Open-Source-and-Contribute or Reinvent-the-wheel-for-L6-promo-and-then-opensource ??

Would the world be better with 10 workflow orchestrator systems or one mature?

Re: Maestro: Netflix's Workflow Orchestrator

#72

Post by bhawks »

delecti wrote: post #60
bluepizza wrote: post #52

Earlier quoted context omitted.

> People need to realize that code is a liability This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun. I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech. Cod…

Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.

Trucks are literally an asset - you can't do depreciation on a liability.

The only way a 'truck' could be a liability is a lease for said truck.

There are plenty of economically rationale reasons why a company may own more trucks they strictly need to manage delivery. For example wanting to handle seasonal bursts, wanting to ensure reliability, preparing for an expansion, being able to lease capacity to other businesses.

Actually you can go replace truck with server and you describe what made AWS make initial sense.

Please stop misusing accounting concepts.

Re: Maestro: Netflix's Workflow Orchestrator

#73

Post by bhawks »

slt2021 wrote: post #70
delecti wrote: post #60

Earlier quoted context omitted.

Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.

For trucking company owning and developing trucks makes sense. But does it make sense for a trucking(streaming) company to create own plumbing equipment? I’d rather use Plumbers Supply Inc that every other company uses from Plumber Depot or use open-source-plumbers.com, because I am not in a plumbing business

The margin on trucking could be so much higher than plumbing that most plumbers could never afford the R&D necessary to advance flushing tech. Big truck operates at a scale where they materially benefit from better flushing, so they take their truck dollars and pour them into their own plumbing lab. Big truck sees this as a competitive advantage that no one else is positioned in the market to unlock. They may one day enter the general plumbing space and disrupt waste management, at their option not obligation of course.

This describes Google and Amazon perfectly - while you can armchair quarterback their biz decisions they are definitely doing well for themselves.

Re: Maestro: Netflix's Workflow Orchestrator

#74

Post by slt2021 »

cortesoft wrote: post #66
slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source t…

So Netflix expects open source community to pick up the maintenance tab ?

I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix:

1. They claim it is distributed system, but it is just a regular Java crud with SQL backend

2. Java-like DSL with parser and classloader (why? Just why?)

Projects like these are the perfect examples of Enterprise Grade FizzBuzz (https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...) and this is exactly what I dont like about it

Re: Maestro: Netflix's Workflow Orchestrator

#75

Post by rjh29 »

cortesoft wrote: post #66
slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source t…

Usually the corporate needs differ too much and they end up keeping their own fork anyway.

Netflix has the resources to maintain this. It's probably more a PR move for their hiring division.

Re: Maestro: Netflix's Workflow Orchestrator

#76

Post by pants2 »

bluepizza wrote: post #52
slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

> People need to realize that code is a liability This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun. I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech. Cod…

Using open-source is a liability too, with added problems of code licensing conflicts, supply chain attacks, zero-day vulnerabilities, relying on maintainers that don’t work for you, etc.

Re: Maestro: Netflix's Workflow Orchestrator

#77

Post by slt2021 »

YawningAngel wrote: post #48
slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

Off-the-shelf open source stuff is often the product of big companies open sourcing internal tools though. Airflow, which you name check, is a great example of this. Temporal is another example in the space. Someone has to be dumb enough to build new stuff

airflow and Temporal has teams dedicated to maintain and extend their system. And these systems are business critical for astronomer/temporal, respectively.

And they develop them in a way that works for many customers and use cases, not just netflix.

But for netflix this is just another auxillary system, out of many others. Just a nice GUI to schedule cron jobs basically, does it make sense to sink resources into custom cron?

Re: Maestro: Netflix's Workflow Orchestrator

#78

Post by thfuran »

ATMLOTTOBEER wrote: post #51
thfuran wrote: post #41

Earlier quoted context omitted.

>When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bullshit When is that, exactly? https://www.statista.com/chart/15692/distribution-of-global-...

“Other” in your diagram is mostly porn

Even supposing that "Other" is just pornhub and nothing else, that's less than one order of magnitude more than Netflix.

Re: Maestro: Netflix's Workflow Orchestrator

#79

Post by cbsmith »

slt2021 wrote: post #74
cortesoft wrote: post #66

Earlier quoted context omitted.

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source t…

So Netflix expects open source community to pick up the maintenance tab ? I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix: 1. They claim it is distributed system, but it is just a regular Java crud with SQL backend 2. Java-like DSL with parser and classloader (why…

> So Netflix expects open source community to pick up the maintenance tab ?

In fairness, the very nature of open source is that the community is only going to pick up the maintenance tab if the value they're getting out of it is worth it.

Re: Maestro: Netflix's Workflow Orchestrator

#80

Post by cbsmith »

bhawks wrote: post #72
delecti wrote: post #60

Earlier quoted context omitted.

Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.

Trucks are literally an asset - you can't do depreciation on a liability. The only way a 'truck' could be a liability is a lease for said truck. There are plenty of economically rationale reasons why a company may own more trucks they strictly need to manage delivery. For example wanting to handle seasonal bursts, wanting to ensure reliability, preparing for an expansion, being able to lease capacity to other busines…

> Please stop misusing accounting concepts.

Assets can also be liabilities. The mortgages in a mortgage backed security is both an asset and a liability, as was only too well demonstrated in 2008... It's an asset in the security portfolio, but until you sell the security, it's a liability for whomever is securitizing it.

Post reply on HN