Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you handle long-running workflows at your company?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 84 posts

Re: Ask HN: How do you handle long-running workflows at your company?

#61

Post by devedlee »

stpedgwdgfhgdd wrote: post #50
devedlee wrote: post #44

We developed and use Argo ( https://github.com/argoproj/argo ), a Kubernetes-native workflow engine. Argo is currently used by companies like Cyrus Biotechnology, Gladly, Google, Intuit, and NVIDIA. Currently collecting use cases and requirements on a Kubernetes-native eventing framework for Argo ( https://github.com/argoproj/argo-events/issues/1 ) to make it easier to kick off workflows.

Does Argo support recovery? In the sense that if a workflow step or the workflow engine crashes halfway, the last (idempotent) action is retried?

The latest Argo release (https://github.com/argoproj/argo/releases) supports resubmitting workflows with "memoized" steps.

Re: Ask HN: How do you handle long-running workflows at your company?

#62

Post by superzamp »

ameyamk wrote: post #49

At LinkedIn we heavily use Azkaban for this. (Open source: https://azkaban.github.io/ ) Azkaban API can be used to launch offline computation jobs as necessary - Azkaban ensures monitoring, SLA alerting, failed restarts and other dependency management etc.

Azkaban really seems to strike the right balance between simplicity and featurefulness, I'll definitely give it a try! Plus it seems relatively simple to deploy & maintain.

The documentation often mention Hadoop and data jobs, have you also used it for non-data things? Would you by chance have some workflows examples?

Re: Ask HN: How do you handle long-running workflows at your company?

#63

Post by superzamp »

geomagilles wrote: post #40
dalore wrote: post #28

Earlier quoted context omitted.

No obvious solutions? Many enterprise companies have a workflow management product. Adobe has one which it makes quite a bit of enterprise revenue from. https://www.adobe.com/uk/marketing-cloud/experience-manager/...

Indeed - but I do not think it's related to the question. The question here is: how do I - as a developer - implement a workflow? Still there are numerous BPM solutions, but often overly sophisticated. You have AWS SWF, but complicated to use, Airflow but in Python only, your own implementation using queues, database, etc... Look at the diversity of answers: there is no obvious answer right now.

From what I see on your website it seems your product indeed has found a sweet spot between business-heavy and deep-tech systems.

The only concern I have is having such a critical part of my application running in a proprietary SaaS environment. Do you have plans to consider on-premise licensing or having an open-source community codebase with enterprise plans?

Re: Ask HN: How do you handle long-running workflows at your company?

#64

Post by jessesuen »

stpedgwdgfhgdd wrote: post #50
devedlee wrote: post #44

We developed and use Argo ( https://github.com/argoproj/argo ), a Kubernetes-native workflow engine. Argo is currently used by companies like Cyrus Biotechnology, Gladly, Google, Intuit, and NVIDIA. Currently collecting use cases and requirements on a Kubernetes-native eventing framework for Argo ( https://github.com/argoproj/argo-events/issues/1 ) to make it easier to kick off workflows.

Does Argo support recovery? In the sense that if a workflow step or the workflow engine crashes halfway, the last (idempotent) action is retried?

I work on argo. The workflow-controller is very tolerant to crashes and designed to be this way. Workflow state is captured in the workflow CRD object (in k8s etcd). Because step names are formulated, in the event of a crash (say before the created pod is persisted in etcd), when the controller restarts and tries to schedule the pod again, it hits an AlreadyExists error and understands how to handle this. Thus, workflows are idempotent in crash scenarios.

Re: Ask HN: How do you handle long-running workflows at your company?

#66

Post by fpierfed »

I worked with long (>> 24 hours, some times up to a week) complex workflows on big (thousands of nodes) clusters. We used custom software layered on top of a job scheduler like PBS Pro or HTCondor. The nice thing about this setup is that it supports re-running failed jobs, has pretty good monitoring, does an OK job at resource selection and allocation and is language agnostic. The last point is good if your workflows have parts written in different languages. There are a handful of conferences a year on these topics by the way. My favorite is HTCondor Week at the University of Wisconsin in Madison. Talks are online [1]

[1]: http://research.cs.wisc.edu/htcondor/HTCondorWeek2017/

Re: Ask HN: How do you handle long-running workflows at your company?

#67

Post by brudgers »

jononor wrote: post #65

What kind of workflows take multiple days? I am assuming that means human inputs are needed for (some) steps?

A debiting and crediting a bank account is an example of a long running workflow...though I am not certain that is what the OP meant. Anyway, as a workflow, the process that maintains an individual account usually runs over many years and perhaps a century or more. The underlying architecture is one reason why banking still (sometimes) uses COBOL...the software was written around abstractions that address the timelines involved. For what it's worth, Michael (not that one) Jackson's Principles of Program Design is where I picked up account balance as a long running process.

Re: Ask HN: How do you handle long-running workflows at your company?

#68

Post by mabn »

jononor wrote: post #65

What kind of workflows take multiple days? I am assuming that means human inputs are needed for (some) steps?

In my case it was waiting until tax information becomes available in another system (IIRC up to several months, but usually few days). Sometimes it was required for a person to actually travel somewhere to get the data via paper forms. Usually human input was needed for some steps but sometimes it was able to complete automatically.

Most waiting (hours-days) happened because the work was waiting in a queue for a user to take care of it.

Re: Ask HN: How do you handle long-running workflows at your company?

#69

Post by ameyamk »

superzamp wrote: post #62
ameyamk wrote: post #49

At LinkedIn we heavily use Azkaban for this. (Open source: https://azkaban.github.io/ ) Azkaban API can be used to launch offline computation jobs as necessary - Azkaban ensures monitoring, SLA alerting, failed restarts and other dependency management etc.

Azkaban really seems to strike the right balance between simplicity and featurefulness, I'll definitely give it a try! Plus it seems relatively simple to deploy & maintain. The documentation often mention Hadoop and data jobs, have you also used it for non-data things? Would you by chance have some workflows examples?

You can use this for any execution. eg. here is a job type to trigger shell command such as ' echo "hello" ' http://azkaban.github.io/azkaban/docs/latest/#command-type

Note execution environment for such jobs is Azkaban executor server itself, so you have to take care of resource management (eg. one job taking all RAM on the machine will affect other jobs running on the same machine)

Re: Ask HN: How do you handle long-running workflows at your company?

#70

Post by inoop »

Have you looked at AWS Step Functions? https://aws.amazon.com/step-functions

edit: to add, I would highly recommend using a workflow engine over a distributed messaging system. With messages it's hard to track where a given work item is in your pipeline, and it's not always easy to do mass operations such as just stopping all running workflows (e.g. when you have an outage) and resuming them later, re-driving failed items from the beginning of the workflow, etc. Workflow engines typically give you a nice dashboard where you can do all those things, for free.

Post reply on HN