Live data from Hacker News

Ask HN: How do you handle long-running workflows at your company?

news.ycombinator.com

1–10 of 84 posts

Ask HN: How do you handle long-running workflows at your company?

#1
The canonical answer to this question apparently used to be ESBs, but the rise of the microservice paradigm eventually pushed them to decline and left a void I'm not sure how is currently filled.

HN, how do you handle your days-long sequences of business steps?

Some seed questions:

* Is your system more P2P or orchestrated?

* Do you leverage some existing tools or built your own?

* Are you confident in your monitoring of errored workflows?

* How do you retry errored workflows?

* If your system if more P2P, how do you keep a holistic view of what's happening? Can you be certain that you don't have any circular event chains?

Re: Ask HN: How do you handle long-running workflows at your company?

#2
In publishing/media:

Some workflows are shorter than others, but in the journalism side the workflows tend to bottom out at a day and max out at a few months (for the workflow, but is ultimately dependent on the weight of the story)...

Most of that is handled above the technology, mind.

The exploration for the right tool(s) is ongoing. I've been leveraged to build one but the status of that clandestine project is in flux to put it lightly. Not sure if I can elaborate on that right now.

Currently, the needs and preferences vary so much that there are many different services used, but the company is seeking to centralize some efforts (like content generation and management) and externalize others (like distribution).

Re: Ask HN: How do you handle long-running workflows at your company?

#3
We basically email stuff around and then when it gets stuck somewhere follow up with another email / conference call to move it forward again. If it keeps getting stuck or doesn't move it's obviously not an important process so it falls out of the loop.

Re: Ask HN: How do you handle long-running workflows at your company?

#5
Our system is based on the Camunda process engine (in a Java EE environment). There's a central process server (or cluster) running the process engine, with events to start process instances.

Workflows are defined using bpmn and then executed by the engine. Errors are reported to the process engine as "Incident", which then show up in the management ui/apis. These can be retried any number of times.

We also have an older system based on Carnot/Stardust/IPP. This one used JMS messages everywhere.

Re: Ask HN: How do you handle long-running workflows at your company?

#6
"State machine" was the easiest for simpler stuff. I put it in quotes because it feels like one, but probably isn't.

Map out each state of your workflow, and having errors give the option to fix immediately, try again, or revert to a previously known-good state. You likely want to start with a 10,000ft view of the workflow and then work on each of those steps as an independent unit and add all of their intermediate steps (on and on until you reach the bottom of the recursion).

This gives you a good opportunity to break things up into microservices that completely handle individual steps if they are big and detailed enough.

P2P is hardest because you will likely need to code something to determine who should decide to move things to the next state (simple majority? one person elected?) and keep track of consensus between all parties.

Orchestration is easier because there's usually one person, one role, or one security claim in control at a particular step and changing who can advance the state at each step is pretty easy as well.

All of this was mostly for the goal of really easy unit testing.

But note that whatever backing data store you use can be changed by any developer unless you code all of the business rules there, too. Many people do not like doing this though because it's not as easy as all of the unit testing frameworks, debuggers, and IDEs we have for code.

The challenge is that you need to know the workflow completely and that will very likely involve talking to a lot of people and the chances you will miss one or two edge cases is high. The counter to that challenge is that as developers building a product that saves time/money, you can bend the workflow to make it easier to code and sometimes eliminate those extra steps (literally, we had someone copying and pasting stuff to 'make it work', so of course we can automate that).

Saving known-good states can also be challenging depending on what you're doing, but if you need change history or diff'ing in a user-consumable form, you'll have to do that anyway. If you get this right, it can save your users a lot of potentially lost work and headache if a bug gets past unit testing.

Once everything is modular, logging isn't too difficult either.

Re: Ask HN: How do you handle long-running workflows at your company?

#7
I've used Redmine with the Checklists plugin for this. Each thing that needs to be done is a redmine issue, and each issue can have a checklist. As team members check off items on the list, the issue logs who/what/when and then the user can assign the next person in the chain to the issue. At the time the checklists plugin didn't include templating functionality (not sure if it does now), so I rolled my own using the Redmine REST api and some PHP.

Hardest part was getting managerial support; they really liked paper.

Re: Ask HN: How do you handle long-running workflows at your company?

#8
We have developed a custom workflow system in PHP for our company (academic text editing and related work). Back then (I was not directly involved from the start) none of the out of the box solutions fit our criteria, and it made more sense to just build a bespoke, custom fitted system. Workflows range from a few days to a month+, with no technical upper limit enforced, as far as I know.

We also don't need a huge throughput, so having something super-optimized was not a large concern.

Re: Ask HN: How do you handle long-running workflows at your company?

#9
post #5

Our system is based on the Camunda process engine (in a Java EE environment). There's a central process server (or cluster) running the process engine, with events to start process instances. Workflows are defined using bpmn and then executed by the engine. Errors are reported to the process engine as "Incident", which then show up in the management ui/apis. These can be retried any number of times. We also have an o…

Interesting, I'm looking into Camunda right now for our processes. How would you describe the experience in terms of adoption and results ?

Re: Ask HN: How do you handle long-running workflows at your company?

#10
I had been looking at using BPMN and an implementation of Camunda as a reasonable goal, but I never found an implementation of running a BPMN service that I liked in the time I had allotted. In the workflow each item is essentially a ticket so you end up with concurrent tickets in the state machine. It also has timers to generate events so you can have that monthly event start and trigger some other actions, and it also includes failure paths.
Post reply on HN