Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

41–50 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#41

Post by thfuran »

ilrwbwrkhv wrote: post #37
hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

Its because Netflix pretends to be a tech company to get the high market cap. So they hire tons of engineers who have nothing to do but rearchitecture the mess their microservices have created. Then there are others who create observability and test harnesses for all of that. When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bulls…

>When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bullshit

When is that, exactly? https://www.statista.com/chart/15692/distribution-of-global-...

Re: Maestro: Netflix's Workflow Orchestrator

#42

Post by ripped_britches »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

100%. Very few times are these systems built as robustly as external folks who earn a profit on building robustness. Best example of course being Stripe. But I see this from everything from visual snapshot testing tools to custom CI workflows. The good thing is you can always rely on competitive market dynamics to price the off the shelf solution down to a reasonable margin above maintenance costs.

Re: Maestro: Netflix's Workflow Orchestrator

#43

Post by ryanianian »

dinobones wrote: post #24
hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

We rolled our own workflow engine and it almost crashed one of our unrelated projects for having so many bugs and being so inflexible. I’m starting to think workflow engines are somewhat of a design smell. It’s enticing to think you can build this reusable thing once and use it for a ton of different workflows, but besides requiring more than one asynchronous step, these workflows have almost nothing in common. Diffe…

> workflow engines are somewhat of a design smell

Probably so, but the real design smell seems to be thinking of a workflow engine as a panacea for sustainable business process automation.

You have to really understand the business flow before you automate it. You have to continuously update your understanding of it as it changes. You have to refactor it into sub-flows or bigger/smaller units of work. You have to have tests, tracer-bullets, and well-defined user-stories that the flows represent.

Else your business flow automation accumulates process debt. Just as much as a full-code-based solution accumulates technical debt.

And, just like technical debt, it's much easier (or at least more interesting) to propose a rewrite or framework change than it is to propose an investment in refactoring, testing, and gradual migrations.

Re: Maestro: Netflix's Workflow Orchestrator

#44

Post by exe34 »

thfuran wrote: post #41
ilrwbwrkhv wrote: post #37

Earlier quoted context omitted.

Its because Netflix pretends to be a tech company to get the high market cap. So they hire tons of engineers who have nothing to do but rearchitecture the mess their microservices have created. Then there are others who create observability and test harnesses for all of that. When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bulls…

>When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bullshit When is that, exactly? https://www.statista.com/chart/15692/distribution-of-global-...

isn't it like 30%?

Re: Maestro: Netflix's Workflow Orchestrator

#45

Post by jefurii »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

This sounds like the beginning of a sales pitch.

Re: Maestro: Netflix's Workflow Orchestrator

#46

Post by stavros »

nijave wrote: post #38
stepanhruda wrote: post #33

Earlier quoted context omitted.

With all due respect, there are so many projects. They don’t care about clashing with a repo that has 12 stars and 14 commits.

Worked at a bank that named their container "cloud" platform GCP and it was in no way related to Google facepalm

Well, if you're so unimaginative as to call your cloud platform " cloud platform", it's not the fault of the second company whose name also starts with a G.

Re: Maestro: Netflix's Workflow Orchestrator

#47

Post by makeset »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

> anything that is not the business critical stuff That's an important qualifier. For skilled teams in performance-critical domains, the inflection point where any outside code becomes a low-quality/low-control liability is not that far.

Re: Maestro: Netflix's Workflow Orchestrator

#48

Post by YawningAngel »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

Off-the-shelf open source stuff is often the product of big companies open sourcing internal tools though. Airflow, which you name check, is a great example of this. Temporal is another example in the space. Someone has to be dumb enough to build new stuff

Re: Maestro: Netflix's Workflow Orchestrator

#49

Post by nijave »

stavros wrote: post #46
nijave wrote: post #38

Earlier quoted context omitted.

Worked at a bank that named their container "cloud" platform GCP and it was in no way related to Google facepalm

Well, if you're so unimaginative as to call your cloud platform " cloud platform", it's not the fault of the second company whose name also starts with a G.

Worse, the G was Gaia (ironically the personification of Earth in Greek mythology). They used "Gaia" as a name for all their internal cloud platforms
Post reply on HN