Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

91–100 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#91

Post by renewiltord »

ilrwbwrkhv wrote: post #37
hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

Its because Netflix pretends to be a tech company to get the high market cap. So they hire tons of engineers who have nothing to do but rearchitecture the mess their microservices have created. Then there are others who create observability and test harnesses for all of that. When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bulls…

> When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bullshit.

That's nothing. My dedicated server delivers two orders of magnitude greater traffic than Pornhub (and everything in the Mindgeek network really). And I don't even need the cloud. Just better engineering.

Re: Maestro: Netflix's Workflow Orchestrator

#92

Post by barnabyjones »

rty32 wrote: post #56
thfuran wrote: post #41

Earlier quoted context omitted.

>When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bullshit When is that, exactly? https://www.statista.com/chart/15692/distribution-of-global-...

What is the methodology of the report? Just one of the questions I have regarding this -- China has nearly 1.4 billion people, and barely any of them use any of the services here. Instead, they have their own video platforms. And you tell me that none of those platforms use at least the same amount of traffic of Prime Video? I doubt it.

I found the report the statistic is from [0]. But note that it says "by app," so I don't think it's actually all traffic, just the top apps. Their reported source is data from 300m customers in different regions.

[0] https://www.sandvine.com/hubfs/Sandvine_Redesign_2019/Downlo...

Re: Maestro: Netflix's Workflow Orchestrator

#93

Post by narism »

slt2021 wrote: post #87
bhawks wrote: post #73

Earlier quoted context omitted.

The margin on trucking could be so much higher than plumbing that most plumbers could never afford the R&D necessary to advance flushing tech. Big truck operates at a scale where they materially benefit from better flushing, so they take their truck dollars and pour them into their own plumbing lab. Big truck sees this as a competitive advantage that no one else is positioned in the market to unlock. They may one day…

Amazon actually steals a lot of open source and repackages it as a “managed AWS service”, they literally deployed managed Airflow as soon as it became popular. The whole aws reinvent is repackaged whatever open source project is trending, hiding control plane from the user and instead expose it via AWS control plane and charge people per usage instead of per server

Can you steal something that is given away freely? People are buying those services so they must be providing some value.

Re: Maestro: Netflix's Workflow Orchestrator

#95

Post by fragmede »

pm90 wrote: post #57
hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

It’s likely because we haven’t yet found a workflow engine/orchestrator thats capable of handling diverse tasks while still being easy to understand and operate. It’s really easy to build a custom workflow engine and optimize it for specific use cases. I think we haven’t yet seen a convergence simply because this tool hasn’t yet been built. Consider the recent rise of tools that quickly dominated their fields: Terraf…

slurm? airflow?

Re: Maestro: Netflix's Workflow Orchestrator

#96

Post by willbeddow »

I'm sure this is very nice, but the article reads as if written by AI. The first thing I'd want to see is an example workflow (both code and configuration) in a realistic use case. Instead, there's a lot of "powerful and flexible" language, but the example workflow doesn't come until halfway down, and then it's just foobar

Re: Maestro: Netflix's Workflow Orchestrator

#97

Post by hulahoof »

slt2021 wrote: post #77
YawningAngel wrote: post #48

Earlier quoted context omitted.

Off-the-shelf open source stuff is often the product of big companies open sourcing internal tools though. Airflow, which you name check, is a great example of this. Temporal is another example in the space. Someone has to be dumb enough to build new stuff

airflow and Temporal has teams dedicated to maintain and extend their system. And these systems are business critical for astronomer/temporal, respectively. And they develop them in a way that works for many customers and use cases, not just netflix. But for netflix this is just another auxillary system, out of many others. Just a nice GUI to schedule cron jobs basically, does it make sense to sink resources into cus…

But when Airbnb created airflow, you could have said the same. It’s just later in its lifecycle.

Re: Maestro: Netflix's Workflow Orchestrator

#98

Post by patmorgan23 »

bhawks wrote: post #72
delecti wrote: post #60

Earlier quoted context omitted.

Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.

Trucks are literally an asset - you can't do depreciation on a liability. The only way a 'truck' could be a liability is a lease for said truck. There are plenty of economically rationale reasons why a company may own more trucks they strictly need to manage delivery. For example wanting to handle seasonal bursts, wanting to ensure reliability, preparing for an expansion, being able to lease capacity to other busines…

A truck also comes with a maintenance liability if you want to continue getting value out of it, just like code.

Re: Maestro: Netflix's Workflow Orchestrator

#99

Post by bushbaba »

slt2021 wrote: post #87
bhawks wrote: post #73

Earlier quoted context omitted.

The margin on trucking could be so much higher than plumbing that most plumbers could never afford the R&D necessary to advance flushing tech. Big truck operates at a scale where they materially benefit from better flushing, so they take their truck dollars and pour them into their own plumbing lab. Big truck sees this as a competitive advantage that no one else is positioned in the market to unlock. They may one day…

Amazon actually steals a lot of open source and repackages it as a “managed AWS service”, they literally deployed managed Airflow as soon as it became popular. The whole aws reinvent is repackaged whatever open source project is trending, hiding control plane from the user and instead expose it via AWS control plane and charge people per usage instead of per server

Accept they also provide the security, billing/invoicing, IaaC, support, provisioning, scaling, list goes on.

As for pay per server vs pay per usage. Heck you know Amazon actually bills the team who caused the cost. And gives finance a report on how much each team is spending and on what. Good luck doing that on prem.

Re: Maestro: Netflix's Workflow Orchestrator

#100

Post by bhawks »

patmorgan23 wrote: post #98
bhawks wrote: post #72

Earlier quoted context omitted.

Trucks are literally an asset - you can't do depreciation on a liability. The only way a 'truck' could be a liability is a lease for said truck. There are plenty of economically rationale reasons why a company may own more trucks they strictly need to manage delivery. For example wanting to handle seasonal bursts, wanting to ensure reliability, preparing for an expansion, being able to lease capacity to other busines…

A truck also comes with a maintenance liability if you want to continue getting value out of it, just like code.

Liabilities are obligations of a company to pay money owed to a lender as a result of a previous transaction.

You are describing an operating expense which has an entirely different nature than a liability.

'comes with a maintenance liability' is a handwaving statement that means practically nothing without a ton of contextual information. A true liability has a contractual set of obligations to pay defined amounts on a agreed upon schedule. No one is going to come after you for not changing the oil on your truck, try missing payments on a lease.

Post reply on HN