Check out Luigi (Python -- https://github.com/spotify/luigi).
I've built (or worked on) a few bespoke systems myself, but Luigi covers better than 80% of what I typically need.
Ask HN: How do you handle long-running workflows at your company?
21–30 of 84 posts
Re: Ask HN: How do you handle long-running workflows at your company?
#22Airflow jobs in the backend.
Re: Ask HN: How do you handle long-running workflows at your company?
#23Nomad just does it for you(mostly):
https://www.nomadproject.io/docs/job-specification/parameter...
Re: Ask HN: How do you handle long-running workflows at your company?
#24Uber wrote its own framework: https://github.com/uber/cadence
Re: Ask HN: How do you handle long-running workflows at your company?
#25We use (as well as sell/implement/customize) an Enterprise Content Management system that has very robust business automation capabilities.
Re: Ask HN: How do you handle long-running workflows at your company?
#26Two options we use, database used as a queue for granular out of process work. If something errors, we'll get a notification for that one record, but the rest will keep processing.
For stuff that we don't need such granularity/replay, we use Amazon's SNS event framework to trigger different APIs.
Sometimes we do a combination of those, an SNS event triggers a lambda that puts a record in the database queue, which gets picked up by a job engine and raises an SNS event that hits an API that sets a record to available.
Re: Ask HN: How do you handle long-running workflows at your company?
#27Check out Luigi (Python -- https://github.com/spotify/luigi ). I've built (or worked on) a few bespoke systems myself, but Luigi covers better than 80% of what I typically need.
Also using luigi here in production since 2015. We use it to manage a multi-day pipeline that is essentially semi-automatic (requires occasional human intervention).
Re: Ask HN: How do you handle long-running workflows at your company?
#28There is no obvious solution right now. That's why we are building Zenaton (I'm cofounder). It's in closed beta by now, but you can have a look at the documentation ( https://zenaton.com/documentation ) and also read some use cases ( https://medium.com/zenaton ). Zenaton provides a very simple way (in your own programming language) to orchestrate background jobs
No obvious solutions? Many enterprise companies have a workflow management product. Adobe has one which it makes quite a bit of enterprise revenue from.
https://www.adobe.com/uk/marketing-cloud/experience-manager/...
Re: Ask HN: How do you handle long-running workflows at your company?
#29We let the web application do it and pray the web server does't croak mid job which it usually does.
Re: Ask HN: How do you handle long-running workflows at your company?
#30I'm the CTO at ProcessMaker and so I might be a little biased. Our customers use our ProcessMaker BPM product if the workflows require human intervention through forms/email other interactions. The reporting tools assist in monitoring and dealing with circular chains.
If you are a developer and want to develop your own system around a workflow engine, we also have www.processmaker.io which is a workflow engine in the cloud. So all the infrastructure hassle is taken care of for you and you communicate via an api to build out your workflows and execute them. I feel like that's better described as an orchestration engine however it supports task assignment to people. An approach like this works well with microservices since it can act as a microservice orchestration engine with a more human workflow approach.
Both of these approaches can be long running (some customers have year long processes running).
Let me know if you want to know more details, happy to share.