Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maestro: Netflix's Workflow Orchestrator

netflixtechblog.com

Post reply on HN
Print view View original

81–90 of 165 posts

Re: Maestro: Netflix's Workflow Orchestrator

#81

Post by cbsmith »

pants2 wrote: post #76
bluepizza wrote: post #52

Earlier quoted context omitted.

> People need to realize that code is a liability This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun. I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech. Cod…

Using open-source is a liability too, with added problems of code licensing conflicts, supply chain attacks, zero-day vulnerabilities, relying on maintainers that don’t work for you, etc.

Not open source is a liability too, with added problems of code licensing conflicts, supply chain attacks, zero-day vulnerabilities, relying on maintainers that don't work for you, etc... ;-)

Re: Maestro: Netflix's Workflow Orchestrator

#82

Post by Nathanba »

hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

It inherently asks for a custom implementation because it's almost like workflows are just how you'd have to code and run everything anyway. Conceptually: why wouldn't we want to reconnect to any work we are currently in progress of, just like in a videogame where if we lose connection for a splitsecond, we want to be able to keep going where we left off? So therefore we must save the current step persistently and make sure that we can resume work and never lose it. Workflow engines also do no magic: They still just run code and if it fails in a place that we didn't manually checkpoint (=by making it into a separate task/workflow/function/action/transaction that is persistable) then we still lose data so.. at that point why not just try doing it this way everywhere whether it's running in a "workflow engine" or not. Before "workflow engines" we already had db transactions but those were mostly for our benefit so we don't mess up the db with partial inserts. Although so far what I've seen in open source workflow engines is that they don't let you work with user input easily, it's sad how they all start a new thread and then just block the thread while it waits for the user to send something. This is obviously not how you'd code a crud operation. In my opinion this is a huge drawback of current workflow engines. If this was solved, we should literally do everything as a workflow I think. Every form submission from the user could offer to let the user continue where he left off and we saved all his data so "he can reconnect to his game" (to revive the videogame metaphor I started with)

Re: Maestro: Netflix's Workflow Orchestrator

#83

Post by cbsmith »

sgloutnikov wrote: post #21
hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

Naming things, cache invalidation, and workflow engines? :) https://github.com/meirwah/awesome-workflow-engines

No, it's just the two things: naming things, cache invalidation, and off by one errors.

Re: Maestro: Netflix's Workflow Orchestrator

#84

Post by saturn8601 »

Anyone here use Activebatch? To me it is the best software I wish had an equivalent for non enterprise users. I have tried and tried to use other "competitors" but Activebatch's simplicity of just attaching a simple MS SQL DB, installing the Windows GUI and execution agent is just click, click, click and now you have a robust GUI based automation environment where you don't have to use code...or if you want, go ahead and use code in any language if you want...but you don't have to.

Airflow may be robust but it is hidden behind a complexity fence that prevents most from seeing whatever its true capability may be. The same goes for other "open source" competitors.

Why can't someone just develop a robust DB backed GUI first system?

I have tried online services as well, they pale in comparison. I guess the cost of maintaining extensions is what kills simpler paid offerings?

Its a complete shame that ActiveBatch is walled off behind a stupid enterprise sales model. This has prevented this wonderful piece of software from being picked up by the wider community. Its like a hidden secret. :/

Re: Maestro: Netflix's Workflow Orchestrator

#85

Post by geodel »

slt2021 wrote: post #74
cortesoft wrote: post #66

Earlier quoted context omitted.

Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore. They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves. How are these open source t…

So Netflix expects open source community to pick up the maintenance tab ? I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix: 1. They claim it is distributed system, but it is just a regular Java crud with SQL backend 2. Java-like DSL with parser and classloader (why…

You are making great points. This is power of Netflix marketing and branding that they are considered as cutting edge tech company. In reality most of Netflix Java projects are pretty mediocre Enterprise Java stuff. Last year or so they have mandated Spring Boot as their development platform for all their web services.

This is exactly same stack I have to deal daily and management reason is it is lowest common denominator that works well with 3-month contract developer to deliver Nth micro service whose sole job is to call another service.

Re: Maestro: Netflix's Workflow Orchestrator

#87

Post by slt2021 »

bhawks wrote: post #73
slt2021 wrote: post #70

Earlier quoted context omitted.

For trucking company owning and developing trucks makes sense. But does it make sense for a trucking(streaming) company to create own plumbing equipment? I’d rather use Plumbers Supply Inc that every other company uses from Plumber Depot or use open-source-plumbers.com, because I am not in a plumbing business

The margin on trucking could be so much higher than plumbing that most plumbers could never afford the R&D necessary to advance flushing tech. Big truck operates at a scale where they materially benefit from better flushing, so they take their truck dollars and pour them into their own plumbing lab. Big truck sees this as a competitive advantage that no one else is positioned in the market to unlock. They may one day…

Amazon actually steals a lot of open source and repackages it as a “managed AWS service”, they literally deployed managed Airflow as soon as it became popular.

The whole aws reinvent is repackaged whatever open source project is trending, hiding control plane from the user and instead expose it via AWS control plane and charge people per usage instead of per server

Re: Maestro: Netflix's Workflow Orchestrator

#88

Post by ldjkfkdsjnv »

slt2021 wrote: post #40

I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability. I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promot…

I was going to say this. I never mess with random libraries like this, always so much pain.

Re: Maestro: Netflix's Workflow Orchestrator

#89

Post by slt2021 »

andbberger wrote: post #63

slightly off topic, but there is dire need for a scientific "workflow manager" built to FAANG engineering standards attuned for the needs of academia (ie primarily designed to facilitate execution of DAGs on clusters). The airflows of the world have complex unnecessary features and require extensive kitbashing to plug into slurm and the academic side of things is a huge mess. Snakemake comes the closest but suffers f…

Academia better to learn k8s and one of the k8s-native workflow orchestrators. This is as close to FAANG grade and open source as they can get, and arguably a bit better than this repo

Re: Maestro: Netflix's Workflow Orchestrator

#90

Post by renewiltord »

hintymad wrote: post #17

I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…

We all have different use-cases. We also have a workflow engine at work but that's because we wanted immediate execution. From submit to execute time can be 100 ms on our system, which makes it also work well for short jobs. Usually, the task coordinator overhead is greater than that on these things.
Post reply on HN