Ask HN: How do you handle long-running workflows at your company?

Firstly, there is an issue of what these tasks are composed of. They tend to start with a human-generated action, result in several programmatic steps interacting with internal and external services. They then tend to result in a write to a private store, or a call to a service that arbitrates this.

For your task initiation: you're basically building a queue even if you don't like queues. I recommend you embrace this where possible. Eventually you may find so much programmatic traffic that a queue will be unsuitable, but that won't change the need for a queue for human-initiated actions. Do try to write task states to a non-durable store so you can watch tasks!

For your task executors: every aspect of them must consider first that any of the sequential actions may fail to execute, and thus cause the entire task to fail. You simply cannot escape the need to retry tasks. Build for this from day one. For inspiration, a primitive but effective system is Amazon SQS. You can achieve similar effects by rerunning blocks in Kafka, and Rabbit has its own solutions. The more heavyweight the mechanics, the more likely the spine of your product is to break at a critical moment. Be careful.

For your microservices, as informed by previous information you must strive for idemopotency on every endpoint. Even if you can't truly reach this (and true provable idempotency is actually very hard), achieving a practical notion of idempotency to accomodate modest retries is absolutely essential. Retrofitting large systems with idempotency is even more difficult than doing it to start with. Accept performance tradeoffs for this without hesitation. Anyone who says that tail latency is more important than data integrity for business logic is either in lottery-winning-rare condition or is over-prioritizing engineering. If a human needs to act to correct bad data the cost of recovery skyrockets and can spiral out of control.

For your final commit stores, remember that they're not infinite or magical and many can't provide very useful concurrency guarantees. Prefer append-only tables even if this obliges you to run cleanup cycles. If you are going to update records in place, try to use stores with "upsert" operations or "test-and-set" mechanics.

Let's loop back around with this advice and answer each of your questions in turn:

> Is your system more P2P or orchestrated?

Orchestrated systems are easier to monitor, understand and build. They tend to run into scaling challenges after a certain level. Write your software to be agnostic to this. Start with orchestrated when possible.

> Do you leverage some existing tools or built your own?

Both. Bespoke workflow tools are easy. Custom, consistent state storage is harder. Shy away from that outside of very special use cases (e.g., integrated CRDTs or a bloom filter for whitelisting events inside a hot loop.

> Are you confident in your monitoring of errored workflows?

Personally: no. It's genuinely difficult to do this. The harder you try, the more likely it is that your error monitoring system becomes the contention point that breaks your system.

> How do you retry errored workflows?

We use SQS to queue workflows. They get a lot of retries by having the queue claw back the message. In some rare cases work times out and is clawed back to the queue spuriously. I've worked hard to make sure all the services that it calls don't care about such cases and result in expensive nops.

> If your system if more P2P, how do you keep a holistic view of what's happening? Can you be certain that you don't have any circular event chains?

The situation is identical for all types of architectures. AS good bit of advice for the later I picked up is NEVER have a workflow fork conditionally into a prior state. Always have them flow downwards and "away" from your event dispatch queues. If you can, use different queues for internal traffic vs external traffic. You might also use different microservices or tags on microsevice requests. All of this is in service of trying to avoid feedback loops in your system.

When the automated system fails, it automatically opens a Jira ticket to get the right people to fix the automated workflow.

You can then use the Jira case history to drive process improvement.