Workflow.py: A Minimalist Workflow Engine • Hacker News Board

Re: Workflow.py: A Minimalist Workflow Engine

#11

Post by jdbernard » Thu, May 31, 2012, 8:19 AM UTC

zobzu wrote: post #4

So someone remade cron+incrond, in python, using more memory, with a more complicated syntax. Okay. I'll pass.

Yeah, but it seems like one of those tools you write because you want it to work your way, not necessarily because it doesn't already exist. I have a few of those: tools that already exist but I wrote my own for fun or because it fit my workflow better.

Also, the Windows version of cron and the like is not as developer-friendly, IMHO. I do not see why this could not work well on both systems.

Re: Workflow.py: A Minimalist Workflow Engine

#12

Post by Morg » Thu, May 31, 2012, 10:55 AM UTC

Indeed the title made the content look shitty.

Workflow engine means an engine to drive workflows - at least for me.

This thing is meh, no reason to use it over existing C tools, I wonder why someone rererererereinvented the (not actually round) wheel.

Re: Workflow.py: A Minimalist Workflow Engine

#13

Post by HerrMonnezza » Thu, May 31, 2012, 12:21 PM UTC

socratic wrote: post #9

Does Python have a BPEL/BPM-style workflow engine like Ruby's ruote? This doesn't seem to be it, but I would love to have a workflow engine which is designed for long running tasks, with periodic human interruption, conditional flows, and so on.

I'm not sure how they compare to Ruby's "ruote", but there are quite a few Python workflow managers and libraries out there.

GC3Pie (http://gc3pie.googlecode.com/) is a Python library for running many-task workflows featuring interruptible execution, interfaces to SGE, PBS and LSF clusters, and composition operators to build dynamic task dependencies (so, not just DAGs). Disclaimer: I'm one of the developers.

Weaver (http://bitbucket.org/pbui/weaver) is a Python front-end for building workflows that can run on top of the Makeflow engine, supporting SGE, Condor and WorkQueue as execution back-ends. (See the comment by its author "pbui" on another HN thread: http://news.ycombinator.com/item?id=4047100)

NiPyPe (http://nipy.sourceforge.net/nipype/) is a Python workflow engine especially targeted at Neuro-Imaging processing (but the core framework is generic, as far as I understand).

I'm pretty sure this list is not exhaustive: many people seem to be re-writing the same core functionality, coming from different fields and/or with slightly different requirements.