Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

github.com

Post reply on HN
Print view View original

41–50 of 69 posts

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#41

Post by asavinov »

reubano wrote: post #36
alexchamberlain wrote: post #34

Earlier quoted context omitted.

This is an implementation of "Flow Based Programming", right? Its a programming paradigm invented before its time IMHO; perfect for a world of streaming data.

> This is an implementation of "Flow Based Programming", right? pretty much.... just without a GUI. My inspiration was method chaining [1] and the first implementation was this [2]. [1] http://martinfowler.com/articles/collection-pipeline/ [2] http://stackoverflow.com/questions/12172934/method-chaining-...

I am currently working on an alternative approach which is based on a column-oriented paradigm for in-stream analytics as opposed to flow-orientation. It is an adaptation of DataCommandr (http://conceptoriented.com) which is a column-oriented approach to data processing. Unfortunately, StreamCommandr is not yet available but it would be interesting to compare advanatages and drawbacks of these two paradigms.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#42

Post by reubano »

greenshackle wrote: post #39
olviko wrote: post #28

Earlier quoted context omitted.

Can’t really remember, sorry, that code was built for processing RSS feeds and data size was never an issue. I will take a look when get some free time… But, I see what you mean. I had to deal with similar issues in commercial projects and the "pull" model (generators in Python ~ "yield return" in C#) almost never a good idea, especially when you have to have concurrent consumers. While callbacks are hard to combine,…

Python 3.5 introduced the async/await syntax, I don't do C# but at a glance it's the same. I've been working on a similar project and I've also found the push model easier.

Care to share your code? I'd like to see a proper push implementation.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#43

Post by reubano »

asavinov wrote: post #41
reubano wrote: post #36

Earlier quoted context omitted.

> This is an implementation of "Flow Based Programming", right? pretty much.... just without a GUI. My inspiration was method chaining [1] and the first implementation was this [2]. [1] http://martinfowler.com/articles/collection-pipeline/ [2] http://stackoverflow.com/questions/12172934/method-chaining-...

I am currently working on an alternative approach which is based on a column-oriented paradigm for in-stream analytics as opposed to flow-orientation. It is an adaptation of DataCommandr ( http://conceptoriented.com ) which is a column-oriented approach to data processing. Unfortunately, StreamCommandr is not yet available but it would be interesting to compare advanatages and drawbacks of these two paradigms.

I didn't see any code examples so it's a bit difficult to figure out what's going on. Are there any open source column-oriented programming implementations?

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#44

Post by alexchamberlain »

reubano wrote: post #36
alexchamberlain wrote: post #34

Earlier quoted context omitted.

This is an implementation of "Flow Based Programming", right? Its a programming paradigm invented before its time IMHO; perfect for a world of streaming data.

> This is an implementation of "Flow Based Programming", right? pretty much.... just without a GUI. My inspiration was method chaining [1] and the first implementation was this [2]. [1] http://martinfowler.com/articles/collection-pipeline/ [2] http://stackoverflow.com/questions/12172934/method-chaining-...

Not sure flow based programming necessitates a GUI any more than object oriented, functional or procedural programming does...

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#45

Post by [deleted] »

reubano wrote: post #36
alexchamberlain wrote: post #34

Earlier quoted context omitted.

This is an implementation of "Flow Based Programming", right? Its a programming paradigm invented before its time IMHO; perfect for a world of streaming data.

> This is an implementation of "Flow Based Programming", right? pretty much.... just without a GUI. My inspiration was method chaining [1] and the first implementation was this [2]. [1] http://martinfowler.com/articles/collection-pipeline/ [2] http://stackoverflow.com/questions/12172934/method-chaining-...

[deleted]

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#46

Post by asavinov »

reubano wrote: post #43
asavinov wrote: post #41

Earlier quoted context omitted.

I am currently working on an alternative approach which is based on a column-oriented paradigm for in-stream analytics as opposed to flow-orientation. It is an adaptation of DataCommandr ( http://conceptoriented.com ) which is a column-oriented approach to data processing. Unfortunately, StreamCommandr is not yet available but it would be interesting to compare advanatages and drawbacks of these two paradigms.

I didn't see any code examples so it's a bit difficult to figure out what's going on. Are there any open source column-oriented programming implementations?

Actually, it is a novel approach so I am not aware of anything similar. Initially, I implemented it for data wrangling (various potentially complex transformations: https://en.wikipedia.org/wiki/Data_wrangling) and it can be found here: https://bitbucket.org/conceptoriented/dce-java. The main point is that instead of defining a flow (graph) of tables we define a flow (graph) of columns. It can be used where Apache Spark is used.

StreamCommandr essentially relies on the same principles and changes only external API and some internal processing. The idea is that any data table is like a stream of records so that we add new records and delete outdated records. Simultaniously, we evaluate other columns and tables by performaing potentially complex computations which are difficult to do in a record flow.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#47

Post by qw »

tudorw wrote: post #10

if someone can spin up a usable gui, charge enough to make a living without compromising on performance, promise some longevity and a way to export of my stuff I would probably pay for that, I loved pipes, the GUI was a big deal for me.

Apache Nifi looks promising. https://nifi.apache.org

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#48

Post by qw »

reubano wrote: post #32
svieira wrote: post #23

Also in this space (and worth looking at for inspiration, especially for other potential sources and sinks of data) - Apache Camel [1]. [1]: http://camel.apache.org/

I don't know if it's bc of the language (java) or something else, but I've never been able to grok apache data projects. I theoretically know what they do, but there's no way I can understand the code, e.g. [1]. [1] http://camel.apache.org/etl-example.html

The code you see in the beginnig is actually code for configuring a "route" from a source of data to a destination. That was for me the key to understanding it. Once you do, you will find the rest of the documentation easier to follow. The earlier versions of Camel put the config in XML filed, but most prefer the DSL approach
Post reply on HN