Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Flow-based Programming

jpaulmorrison.com

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Flow-based Programming

#11

Post by amelius »

tbirdz wrote: post #6

On the topic of dataflow programming I found the book "Dataflow and Reactive Programming Systems: A Practical Guide to Developing Dataflow and Reactive Programming Systems" by Matt Carkci [0] to give a pretty good overview of the different types and implementations of dataflow programming systems. I think that flow-based programming sounds incredibly cool, but I haven't found any project where I could use it well...…

I think the biggest problems with flow based or reactive programming are the occurrence of glitches, and, more importantly, the problem of how to compute updates without performing unnecessary work.

I wonder if these problems have been successfully addressed somewhere.

Re: Flow-based Programming

#12

Post by corysama »

tbirdz wrote: post #8
samuell wrote: post #7

Earlier quoted context omitted.

Matt Carkci's book discusses the different closely related topics of data flow, flow based programming and reactive programming, very clearly and i quite some detail: http://dataflowbook.com/cms Can highly recommend.

Wow, we just recommended the same book within seconds of each other. But seriously it's a good book, if you have interest in the topic it's worth reading.

I believe the author started http://www.reddit.com/r/dataflowprogramming/ It's a bit slow, but there's a lot of good stuff in there already.

Re: Flow-based Programming

#14

Post by harlowja »

A framework that is similar to this in python (that I am one of the authors of) [1] (and [2] for more in-depth docs/examples). It's used in openstack (which some people may have heard of) where appropriate (and where desirable...) to help in consistent, scalable execution using a flow like methodology to declaratively describe workflows (and execute them using various strategies/engines).

[1] https://github.com/openstack/taskflow

[2] http://docs.openstack.org/developer/taskflow/

Re: Flow-based Programming

#15

Post by jpaulmorrison »

There's a good discussion in the Google group of the ideas that differentiate what we are now calling "classical" FBP (or CFBP) from FBP-like systems, more and more of which are appearing all the time - in particular the topic "Underlying models" - https://groups.google.com/forum/#!topic/flow-based-programmi... .

If you are interested in the more large volume, data-processing, "classical" FBP implementations, a good place to start is my book, "Flow-Based Programming", 2nd ed., and the web site http://www.jpaulmorrison.com/fbp/ .

Re: Flow-based Programming

#16

Post by samuell »

harlowja wrote: post #14

A framework that is similar to this in python (that I am one of the authors of) [1] (and [2] for more in-depth docs/examples). It's used in openstack (which some people may have heard of) where appropriate (and where desirable...) to help in consistent, scalable execution using a flow like methodology to declaratively describe workflows (and execute them using various strategies/engines). [1] https://github.com/opens…

The one I did know in python was PaPy:

https://code.google.com/p/papy/

Will have to have a closer look at this one as well, though.

Re: Flow-based Programming

#18

Post by harlowja »

samuell wrote: post #16
harlowja wrote: post #14

A framework that is similar to this in python (that I am one of the authors of) [1] (and [2] for more in-depth docs/examples). It's used in openstack (which some people may have heard of) where appropriate (and where desirable...) to help in consistent, scalable execution using a flow like methodology to declaratively describe workflows (and execute them using various strategies/engines). [1] https://github.com/opens…

The one I did know in python was PaPy: https://code.google.com/p/papy/ Will have to have a closer look at this one as well, though.

Cool, looks similarish; feel free to jump on freenode (usually during weekdays) and ping me, channel #openstack-state-management or #openstack-oslo

Pretty diagram also @ https://wiki.openstack.org/wiki/TaskFlow#Big_picture ;)

Re: Flow-based Programming

#20

Post by seanmcdirmid »

amelius wrote: post #11
tbirdz wrote: post #6

On the topic of dataflow programming I found the book "Dataflow and Reactive Programming Systems: A Practical Guide to Developing Dataflow and Reactive Programming Systems" by Matt Carkci [0] to give a pretty good overview of the different types and implementations of dataflow programming systems. I think that flow-based programming sounds incredibly cool, but I haven't found any project where I could use it well...…

I think the biggest problems with flow based or reactive programming are the occurrence of glitches, and, more importantly, the problem of how to compute updates without performing unnecessary work. I wonder if these problems have been successfully addressed somewhere.

My work [1] addresses glitches head on (hence the system is called "Glitch") by tolerating glitches and at least making the extra work functionally benign (via logging and rollback, like a transaction). Glitch is not based on data flow, however, but somewhere in between even if still very reactive.

I've built a few data flow systems in the past (my dissertation work [2] was based on one), but I decided to tolerate rather than avoid glitches early on because I found that way to be more resilient. Flapjax [3] avoids glitching via a topological sort, which I rather see as an optimization (to reduce unnecessary work) rather than as a correctness issue.

[1] http://research.microsoft.com/apps/pubs/default.aspx?id=2265...

[2] http://research.microsoft.com/pubs/179366/mcdirmid06superglu...

[3] http://cs.brown.edu/~sk/Publications/Papers/Published/mgbcgb...

Post reply on HN