Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

61–70 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#61

Post by skissane »

I'm a bit confused about what is going on here: This project appears to use Netflix/conductor [0]. But you go to that repo, you see it has been archived, with a message saying it is replaced by Netflix's internal non-OSS version, and by unmentioned community forks – by which I assume they mean Orkes Conductor [1]. But this isn't using Orkes Conductor, it looks like it is using the discontinued Netflix version `com.netflix.conductor:conductor-core:2.31.5` [2] – and an outdated version of it too.

[0] https://github.com/Netflix/conductor

[1] https://github.com/conductor-oss/conductor

[2] https://github.com/Netflix/maestro/blob/e8bee3f1625d3f31d84d...

Re: Maestro: Netflix's Workflow Orchestrator

#62

Post by halamadrid »

skissane wrote: post #61

I'm a bit confused about what is going on here: This project appears to use Netflix/conductor [0]. But you go to that repo, you see it has been archived, with a message saying it is replaced by Netflix's internal non-OSS version, and by unmentioned community forks – by which I assume they mean Orkes Conductor [1]. But this isn't using Orkes Conductor, it looks like it is using the discontinued Netflix version `com.ne…

Yes Netflix abandoned Conductor long time ago [0]. The other repo is built and managed by Orkes after Netflix abandoned it.

[0] https://techcrunch.com/2023/12/13/orkes-forks-conductor-as-n...

Re: Maestro: Netflix's Workflow Orchestrator

#63

Post by andbberger »

slightly off topic, but there is dire need for a scientific "workflow manager" built to FAANG engineering standards attuned for the needs of academia (ie primarily designed to facilitate execution of DAGs on clusters). The airflows of the world have complex unnecessary features and require extensive kitbashing to plug into slurm and the academic side of things is a huge mess. Snakemake comes the closest but suffers from massive feature creep, a bizarre specification DSL (superset of python) and blurred resource requirement abstraction boundaries.

Re: Maestro: Netflix's Workflow Orchestrator

#64

Post by jdmichal »

nijave wrote: post #49
stavros wrote: post #46

Earlier quoted context omitted.

Well, if you're so unimaginative as to call your cloud platform " cloud platform", it's not the fault of the second company whose name also starts with a G.

Worse, the G was Gaia (ironically the personification of Earth in Greek mythology). They used "Gaia" as a name for all their internal cloud platforms

Hello fellow ex-employee of that bank. I was in a segment governed by PCI, and they wouldn't even let us touch Gaia in fear of the whole thing being declared in scope

Re: Maestro: Netflix's Workflow Orchestrator

#65

Post by tempest_ »

ilrwbwrkhv wrote: post #37
hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

Its because Netflix pretends to be a tech company to get the high market cap. So they hire tons of engineers who have nothing to do but rearchitecture the mess their microservices have created. Then there are others who create observability and test harnesses for all of that. When Pornhub and other porn sites can deliver orders of magnitude more data across the world with much simpler systems, you know it's all bulls…

To be fair when netflix started they were solving legitimate problems that a major streaming provider would have.

In the time since those problems have been solved and now are offered as a service by most cloud providers (for a hefty fee of course)

Re: Maestro: Netflix's Workflow Orchestrator

#66

Post by cortesoft »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore.

They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves.

How are these open source tools supposed to be created in the first place? This is the process, someone has to do it

Re: Maestro: Netflix's Workflow Orchestrator

#67

Post by bilalq »

delecti wrote: post #60
bluepizza wrote: post #52

Earlier quoted context omitted.

> People need to realize that code is a liability This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun. I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech. Cod…

Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.

But thinking of those trucks primarily as a liability is exactly the kind of mindset that leads to companies minimizing their liabilities instead of maximizing their potential.

Re: Maestro: Netflix's Workflow Orchestrator

#68

Post by skissane »

halamadrid wrote: post #62
skissane wrote: post #61

I'm a bit confused about what is going on here: This project appears to use Netflix/conductor [0]. But you go to that repo, you see it has been archived, with a message saying it is replaced by Netflix's internal non-OSS version, and by unmentioned community forks – by which I assume they mean Orkes Conductor [1]. But this isn't using Orkes Conductor, it looks like it is using the discontinued Netflix version `com.ne…

Yes Netflix abandoned Conductor long time ago [0]. The other repo is built and managed by Orkes after Netflix abandoned it. [0] https://techcrunch.com/2023/12/13/orkes-forks-conductor-as-n...

I haven't touched Conductor for a few years now, but back in 2020 I did some work trying to implement it, even submitted a few PRs – https://github.com/Netflix/conductor/pulls?q=is%3Apr+author%...

My impression of the code base, is I felt like it needed a lot of work to run in a non-Netflix environment. Which is part of why the project I was working on ended up abandoning Conductor – we were going to embed Conductor in our product as a workflow engine, we ended up building our own workflow engine from scratch instead. Another team did end up using it for some internal use cases, but scalability/reliability/etc are less of a concern for internal use cases as opposed to customer-facing ones.

And then Netflix abandons it – and then they open source something else which depends on an old version of it – well, I'm happy they open source anything, but it fits with my earlier impression – throwing stuff over the fence which can be a struggle to adopt in an outside environment. Still, throwing it over the fence is better than not releasing it at all.

Re: Maestro: Netflix's Workflow Orchestrator

#69

Post by shermantanktop »

bilalq wrote: post #67
delecti wrote: post #60

Earlier quoted context omitted.

Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.

But thinking of those trucks primarily as a liability is exactly the kind of mindset that leads to companies minimizing their liabilities instead of maximizing their potential.

Especially when the cost of minimizing (long hours, unsafe conditions) is not felt by decision makers, and may not materialize for a while, but the benefits of maximizing their potential is felt directly and immediately.

Incentives are everything. That's why managers are so careful when applying them to their own jobs.

Re: Maestro: Netflix's Workflow Orchestrator

#70

Post by slt2021 »

delecti wrote: post #60
bluepizza wrote: post #52

Earlier quoted context omitted.

> People need to realize that code is a liability This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun. I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech. Cod…

Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.

For trucking company owning and developing trucks makes sense.

But does it make sense for a trucking(streaming) company to create own plumbing equipment? I’d rather use Plumbers Supply Inc that every other company uses from Plumber Depot or use open-source-plumbers.com, because I am not in a plumbing business

Post reply on HN