We have a lot of long running workflows written in DTF
Ask HN: How do you handle long-running workflows at your company?
81–84 of 84 posts
Re: Ask HN: How do you handle long-running workflows at your company?
#82I'm going into get-off-my-lawn mode here if you don't mind. I don't see why this requires a new-fangled technology or buzzword. Just have a status code(s) or indicator(s) on a given request. The client side or requesting service(s) can periodically check on the status using polling and/or user status update requests. For example, poll automatically every 2 minutes (to avoid flooding the network), but give user the op…
The benefits of the frameworks become more tangible as the structure of your workflows become more complex and with (acyclic) dependencies and with distributed execution.
True, but what if the "grow complex" step doesn't happen? YAGNI. If you are in a domain that needs complex work-flows, I can see selecting a work-flow framework. But I've seen some really ugly frameworks where one put in Cadillacs for every part when Chevys would do just fine 98% of the time. Staff have to learn, understand, maintain, and tune complex frameworks.
Re: Ask HN: How do you handle long-running workflows at your company?
#83Earlier quoted context omitted.
The benefits of the frameworks become more tangible as the structure of your workflows become more complex and with (acyclic) dependencies and with distributed execution.
True, but what if the "grow complex" step doesn't happen? YAGNI. If you are in a domain that needs complex work-flows, I can see selecting a work-flow framework. But I've seen some really ugly frameworks where one put in Cadillacs for every part when Chevys would do just fine 98% of the time. Staff have to learn, understand, maintain, and tune complex frameworks.
I understand the concern. It's kind of like using a web framework — you probably wouldn't write your own form processing and ORM code to avoid injection attacks when it's already been built. Airflow has analogous features and protections for tasks/workflows.
Most companies adopting Airflow already have workflow requirements like this, even if it's just a single transform or moving data from one system to another.
Even if you have just a two-step workflow with Task B dependent upon success of Task A, Airflow offers protection, historical stats, email alerting, etc over trying to schedule successive cron jobs with built-in assumptions, hacking together a dependency system, etc.
To me, Airflow is the Honda of this domain. Overall it's a relatively small and simple framework from the DAG author's perspective.
Re: Ask HN: How do you handle long-running workflows at your company?
#84The last time I did it maybe a year ago, I had a combination of batch jobs that once completed, should kick off other batch jobs.
I created a poor mans fire and forget pub/sub model where when one process was finish it would "raise an event".
Raising an event, would look in Hashicorp's Consul to see what jobs should be run based on the event and submit a job to Nomad. There were a number of EC2 instances running Nomad agents that would kick off the subsequent jobs. Nomad jobs could be executables or Docker containers.
I was very much a Hashicorp fanboy until I transitioned to using native AWS services. These days I would probably use AWS Step Functions.