Does anyone have recommendation for a non-software dev environment? e.g. user task workflow management.
I'm looking for a fully fledged product and/or an API backend.
Ask HN: How do you handle long-running workflows at your company?
51–60 of 84 posts
Re: Ask HN: How do you handle long-running workflows at your company?
#52https://concourse-ci.org/ is extremely flexible and we use it for a number of complex workflows
Re: Ask HN: How do you handle long-running workflows at your company?
#53In one of my previous companies, they used Airflow(by airbnb) to schedule and manage workflows. Previously they were using nothing but CRON. It turned out to be not so efficient to retry failed workflows and cancel the execution of following dependent jobs. Airflow turned out to be a great fit for our case. I highly recommend checking it out.
We're using Apache Airflow [2] internally as well. It's pretty featureful and addresses most concerns mentioned by OP, such as orchestration, retries, open source code reuse, and dependency management. It has primitive monitoring and alerting, but one needs to bring something external for that today.
Shameless plug - My startup [2] offers Airflow as a SaaS as well as an enterprise distribution with monitoring tools to build upon core Airflow.
Re: Ask HN: How do you handle long-running workflows at your company?
#54In my previous job (banking) we ware using Appian for all our workflows. It was a strange beast of a Java UI application with a k/kdb core and database.
It's UI was really good, much better than Activiti and similar BPM systems; you could create a rather complex workflow with almost no code, just by creating yout BPMN flow through the built-in editor. Also, the editor and the rules system was builtin on the web UI so you didn't have too use external, eclips-ish tools unless you wanted to write custom BPMN nodes, mainly for integration with external systems. Errors and retries were handled through BPMN.
The main problem was the K core: Because nobody knew how to write K we relied on the Java API for access to the kdb database (actually messing with the kdb directly was not even supported by Appian thus even if somebody was willing to learn the bank wouldn't let him mess with the kdbs), which, because of the restrictions it had resulted to having to manually edit a couple of hundred live process instances to change a task assignee or skip a non working custom node... Also because kdbs are stored in memory we needed a very big amount of RAM on the server; which was growing larger proportionally with the ptocess instances.
Even with these shortcomings, I still think that it was a good product, much better than other workflow solutions like Activiti or jbpm or Alfresco. One last thing: Appian was way too expensive; don't consider it if you are not a bank...
Re: Ask HN: How do you handle long-running workflows at your company?
#55Check out Luigi (Python -- https://github.com/spotify/luigi ). I've built (or worked on) a few bespoke systems myself, but Luigi covers better than 80% of what I typically need.
There was a good tutorial at PyCon this past weekend called Workflow Engines Up and Running [1] on Python workflow automation frameworks, specifically comparing Luigi vs Airflow. The video is on YouTube as well [2].
Re: Ask HN: How do you handle long-running workflows at your company?
#56I'm going into get-off-my-lawn mode here if you don't mind. I don't see why this requires a new-fangled technology or buzzword. Just have a status code(s) or indicator(s) on a given request. The client side or requesting service(s) can periodically check on the status using polling and/or user status update requests. For example, poll automatically every 2 minutes (to avoid flooding the network), but give user the op…
The benefits of the frameworks become more tangible as the structure of your workflows become more complex and with (acyclic) dependencies and with distributed execution.
Re: Ask HN: How do you handle long-running workflows at your company?
#57Does anyone have recommendation for a non-software dev environment? e.g. user task workflow management. I'm looking for a fully fledged product and/or an API backend.
Does something like Zapier fit your use cases?
Re: Ask HN: How do you handle long-running workflows at your company?
#58A custom layer built on top of Celery that allows for better monitoring and dependency management, amongst other things. Monitoring, particularly of failure is pretty ok in Celery anyway. The whole thing can generate its own graph by inspecting dependencies, and we use dagre to draw pretty process workflows with status, interactions and monitoring.
Any chance your layer is open source? I'd be curious to see how it compares to something like Airflow with a Celery executor.
Re: Ask HN: How do you handle long-running workflows at your company?
#59Earlier quoted context omitted.
No obvious solutions? Many enterprise companies have a workflow management product. Adobe has one which it makes quite a bit of enterprise revenue from. https://www.adobe.com/uk/marketing-cloud/experience-manager/...
Indeed - but I do not think it's related to the question. The question here is: how do I - as a developer - implement a workflow? Still there are numerous BPM solutions, but often overly sophisticated. You have AWS SWF, but complicated to use, Airflow but in Python only, your own implementation using queues, database, etc... Look at the diversity of answers: there is no obvious answer right now.
Airflow can run tasks written in languages besides Python in several ways, such as through the BashOperator, DockerOperator, dispatching a job to a Spark cluster, etc. It's common to use multiple languages.
It's really just the configuration for tasks, DAGs, etc that must be done in Python. I know some people have even automating that to pull from yaml or json instead, but I prefer to have the flexibility myself.
Re: Ask HN: How do you handle long-running workflows at your company?
#60* Orchestrated, in-house built workflow execution engine.
* Message queue based with service listeners that translate and dispatch messages to individual workers via HTTP requests.
* Workflow execution state is currently backed by RDBMS.
* Infrastructure errors with workflow executions are exceedingly rare and devops can push a button to retry a step if they failed due to a transient condition.
Now, the important bit:
* Retries due to business logic error aren't really a thing unless there is a defined recovery transition for that step in the workflow. This forces people to acknowledge their code did something unexpected (or the workflow definition itself doesn't properly handle all necessary error cases) and fix the underlying issue. Once the root cause is identified and fixed, the workflow instance can be canceled and resubmitted. However, since things that do work usually have side effects, there is sometimes manual cleanup that has to occur that falls on the development team to fix (with assistance from the devops team, if needed). No one likes doing cleanups or getting on devops bad side so there is an incentive to make sure code and workflows are well tested before being released to production.