21–30 of 165 posts
I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…
why would one consider this over something more established such as Temporal, also I see Maestro is written in Java vs Temporal's Go
why would one consider this over something more established such as Temporal, also I see Maestro is written in Java vs Temporal's Go
Netflix also uses temporal: https://temporal.io/in-use/netflix
(website doesn't resolve for me)
EDIT: I found the GitHub page
I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines. Update: I just find it really interesting that many individuals in many companies lik…
I’m starting to think workflow engines are somewhat of a design smell.
It’s enticing to think you can build this reusable thing once and use it for a ton of different workflows, but besides requiring more than one asynchronous step, these workflows have almost nothing in common.
Different data, different APIs, different feedback required from users or other systems to continue.
Whats the difference of this and enqueue work into a queue then waiting for a job to pick it up at a scheduled time? Not saying build a Kafka cluster to serve this but most cloud providers have queuing tools.
Dependencies: what can be done in parallel and what must be done in sequence? For example, three tasks get pushed in the queue and only after all three finish a fourth task must be run.
Retries: The concept is simple. The details are killer. For example, ifa task fails, how long should the delay between retries be? Too short and you create a retry storm. Forget to add some jitter and you get thundering hoards all retrying at the same time.
Scheduling: Because cron is good enough, until it isn't.
A good workflow solution provides battle tested versions of all of the above. Better yet, a great workflow solution makes it easier to keep business logic separate from plumbing so that it's easier to reason about and test.
Whats the difference of this and enqueue work into a queue then waiting for a job to pick it up at a scheduled time? Not saying build a Kafka cluster to serve this but most cloud providers have queuing tools.
a lot more than just e.g. celery jobs
Earlier quoted context omitted.
Netflix also uses temporal: https://temporal.io/in-use/netflix
Is Temporal still alive? (website doesn't resolve for me) EDIT: I found the GitHub page https://github.com/temporalio/temporal
Is this meaningfully different from Conductor (which they archived a while back)? Browsing through the code I see quite a few similarities. Plus the use of JSON as the workflow definition language.
https://github.com/Netflix/maestro/blob/main/maestro-engine/...
https://netflixtechblog.com/orchestrating-data-ml-workflows-...
Very nice, Netflix has a reputation of making great OSS products. I wonder where does this stand with Conductor.
https://netflixtechblog.com/orchestrating-data-ml-workflows-...
https://github.com/Netflix/maestro/blob/main/maestro-engine/...
why would one consider this over something more established such as Temporal, also I see Maestro is written in Java vs Temporal's Go