Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you handle long-running workflows at your company?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 84 posts

Re: Ask HN: How do you handle long-running workflows at your company?

#71

Post by jodison »

We use Apache Oozie (http://oozie.apache.org/) an orchestration system for Hadoop. We don't run days-long workflows, but we run some that have over a dozen steps, and I have no reason to believe Oozie couldn't handle longer-running workflows. Oozie has facilities for handling retries based on user-defined behaviors, and because it can run shell scrips, Java apps, Spark jobs, and most anything in the Hadoop ecosystem, I've found it to be pretty easy to integrate with our other tooling. My one complaint (and it's more a complaint with YARN) is that it can be quite difficult to get your hands on logs when your workflows fail. You can get them, but it can be a real pain.

We were running Oozie on Cloudera, but are migrating to AWS, and I was pleased to find that it can be installed on an EMR cluster[1] and managed with Hue[2] which has a decent UI to administer the schedule with, and a visualization depicting the workflow DAG.

[1]: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-conf...

[1]: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-oozi...

[2]: http://gethue.com/tutorial-a-new-ui-for-oozie/

Re: Ask HN: How do you handle long-running workflows at your company?

#72

Post by berlinda428 »

Think about what divulging that feeling might mean to your relationship. It implies that you do not trust him; what if you’re wrong? How will he feel? How will you feel? You need to keep this in mind for every step that follows 4, below.Figure out your next steps if step 5 ‘fails’ (you feel as though you did not get a truthful answer, or for some other reason).Ask. Not on the way to work, or in the bathroom as you’re blow-drying your hair; find a good time when you are able to both talk openly, and ask him.Tell him why you are asking, and how you feel.If you feel you need to dig further do so. Remember, you are damaging a trusting relationship every time you decide to take it any further, but it is entirely up to you.enigmahacker42@gmail.com can help you find out easily. he will some all your problems related to spying and getting information hack anything.Text +1 2034396340

Re: Ask HN: How do you handle long-running workflows at your company?

#73

Post by NewDimension »

tedmiston wrote: post #57
NewDimension wrote: post #51

Does anyone have recommendation for a non-software dev environment? e.g. user task workflow management. I'm looking for a fully fledged product and/or an API backend.

Does something like Zapier fit your use cases?

Zapier looks like a app trigger. I'm looking for more of a task management system.

Re: Ask HN: How do you handle long-running workflows at your company?

#74

Post by aprdm »

In the past used both Celery and RabbitMQ for some custom services... however we also used to use Qube (A render farm manager) to link long running tasks together in a dependency chain which was workflow specific.

Some times job could take days to finish (doing a water simulation in a 4k res sequence)

Qube has resource constraints per job as well as number of tries and so on. Those would all be configured at job (think workflow) submission time.

Re: Ask HN: How do you handle long-running workflows at your company?

#75

Post by slucha »

inoop wrote: post #70

Have you looked at AWS Step Functions? https://aws.amazon.com/step-functions edit: to add, I would highly recommend using a workflow engine over a distributed messaging system. With messages it's hard to track where a given work item is in your pipeline, and it's not always easy to do mass operations such as just stopping all running workflows (e.g. when you have an outage) and resuming them later, re-driving failed…

Do you have some recommendations for workflow engines?

Re: Ask HN: How do you handle long-running workflows at your company?

#76

Post by kiechu »

andscoop wrote: post #11

The only tool I have found that checks all those boxes (and then some) is Airflow. I liked it so much that I went to work for Astronomer.io, which is building a managed and on-prem solution to Airflow. It's not the perfect tool, but we are striving to make it better.

Thumbs up for Airflow. It's is great for ETL tasks.

Re: Ask HN: How do you handle long-running workflows at your company?

#77

Post by geomagilles »

superzamp wrote: post #63
geomagilles wrote: post #40

Earlier quoted context omitted.

Indeed - but I do not think it's related to the question. The question here is: how do I - as a developer - implement a workflow? Still there are numerous BPM solutions, but often overly sophisticated. You have AWS SWF, but complicated to use, Airflow but in Python only, your own implementation using queues, database, etc... Look at the diversity of answers: there is no obvious answer right now.

From what I see on your website it seems your product indeed has found a sweet spot between business-heavy and deep-tech systems. The only concern I have is having such a critical part of my application running in a proprietary SaaS environment. Do you have plans to consider on-premise licensing or having an open-source community codebase with enterprise plans?

Thx. I totally understand your concerns. We work hard to make developer life much more easy. That's also why the solution is hosted. So you do not have to install, maintain, scale your own system. Just to clarify (if needed): your tasks are executed on your servers, we handle only the orchestration itself. Pricing is a work in progress, but we will probably offer a large free usage.

Re: Ask HN: How do you handle long-running workflows at your company?

#79

Post by BMarkmann »

NewDimension wrote: post #73
tedmiston wrote: post #57

Earlier quoted context omitted.

Does something like Zapier fit your use cases?

Zapier looks like a app trigger. I'm looking for more of a task management system.

Check out the BPM tools others have mentioned (Activiti, Camunda, etc...)

Re: Ask HN: How do you handle long-running workflows at your company?

#80

Post by probledo »

xemdetia wrote: post #10

I had been looking at using BPMN and an implementation of Camunda as a reasonable goal, but I never found an implementation of running a BPMN service that I liked in the time I had allotted. In the workflow each item is essentially a ticket so you end up with concurrent tickets in the state machine. It also has timers to generate events so you can have that monthly event start and trigger some other actions, and it a…

So what is your problem using BPMN?
Post reply on HN