Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you handle long-running workflows at your company?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 84 posts

Re: Ask HN: How do you handle long-running workflows at your company?

#11

Post by andscoop »

The only tool I have found that checks all those boxes (and then some) is Airflow. I liked it so much that I went to work for Astronomer.io, which is building a managed and on-prem solution to Airflow.

It's not the perfect tool, but we are striving to make it better.

Re: Ask HN: How do you handle long-running workflows at your company?

#12

Post by mirceal »

charriu wrote: post #5

Our system is based on the Camunda process engine (in a Java EE environment). There's a central process server (or cluster) running the process engine, with events to start process instances. Workflows are defined using bpmn and then executed by the engine. Errors are reported to the process engine as "Incident", which then show up in the management ui/apis. These can be retried any number of times. We also have an o…

Multiple things: 1) Camunda w/ a Postgres RDS DB. Works for more complex stuff that’s expressed in BPMN 2) If the workflow involves mostly automated stuff and is not running for years, AWS SWF (usually coupled with an API for checkpointing state, keeping track of wflows)

Re: Ask HN: How do you handle long-running workflows at your company?

#13

Post by plcancel »

IIS, AppFabric, Windows Workflow Foundation services (WF). Leverage there for orchestration, persistence, error handling, etc. Considering the demise of AppFabric, do you mind if I Ask HN: How would you handle these long-running workflows in the long-run (and keep IIS and WF)?

Re: Ask HN: How do you handle long-running workflows at your company?

#15

Post by fredley »

A custom layer built on top of Celery that allows for better monitoring and dependency management, amongst other things. Monitoring, particularly of failure is pretty ok in Celery anyway.

The whole thing can generate its own graph by inspecting dependencies, and we use dagre to draw pretty process workflows with status, interactions and monitoring.

Re: Ask HN: How do you handle long-running workflows at your company?

#16

Post by agotterer »

On some of our Ruby workflows We use Sidekiq Pro which has scheduled and batched jobs. The batched jobs is neat because it has a callback feature that you can use for starting additional steps / workflows. We monitor/alert on progress with statsd, datadog, and the sidekiq ui.

Re: Ask HN: How do you handle long-running workflows at your company?

#17

Post by vorpalhex »

Our main bus for microservices is a RabbitMQ cluster. Most services have their own isolated write store and read store (which might be a true read store, or just a db replica).

Long running jobs are a rarity, so we usually spin up a new RabbitMQ cluster and services, but tie those services back to the main write/read stores. This allows regular operations to still occur, but we can monitor the bulk process and commit resources to it in a more isolated fashion.

Errors end up in error queues in Rabbit, and can be dumped back in to be reprocessed if appropriate (or just ignored if it's a side effect we don't care about).

Once it's setup and running, it works well enough. Spinning up a new rabbit cluster and service instances is currently manual, but since we've moved to Kubernetes I'm hoping this can be automated almost entirely.

Re: Ask HN: How do you handle long-running workflows at your company?

#20

Post by geomagilles »

There is no obvious solution right now. That's why we are building Zenaton (I'm cofounder). It's in closed beta by now, but you can have a look at the documentation (https://zenaton.com/documentation) and also read some use cases (https://medium.com/zenaton). Zenaton provides a very simple way (in your own programming language) to orchestrate background jobs
Post reply on HN