Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

github.com

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#31

Post by olviko »

rubber_duck wrote: post #27
olviko wrote: post #20

Earlier quoted context omitted.

Nice project. I wrote something similar in C# long time ago [1]. Mostly to monitor job feeds and craigslist [2] :-) It supports RSS and Atom, async, various filters, deduplication, etc Yahoo Pipes was a nice project, but as its popularity grew, it started getting blocked more and more. It was also hard to build and maintain pipelines with more than a few steps. [1] https://github.com/olviko/RssPercolator [2] https://…

Maybe I'm not seeing something but wouldn't this be simpler with http://reactivex.io/ (Rx.NET specifically) ? Just implement IObservable that pushes RSS events and then use Rx Filter/Select/Merge/GroupBy to filter/join/synchronize/whatever

Maybe. Don't think any of this existed when I need it...

1. The code is tiny with 90% of it dealing with RSS parsing and filtering. Using RX.NET wouldn't really simplify anything.

2. I wanted a library that I can integrate into my apps and run locally to avoid throttling, robots.txt and other BS Yahoo Pipes was suffering from.

I am also not a huge fan of RX... to put it mildly

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#32

Post by reubano »

svieira wrote: post #23

Also in this space (and worth looking at for inspiration, especially for other potential sources and sinks of data) - Apache Camel [1]. [1]: http://camel.apache.org/

I don't know if it's bc of the language (java) or something else, but I've never been able to grok apache data projects. I theoretically know what they do, but there's no way I can understand the code, e.g. [1].

[1] http://camel.apache.org/etl-example.html

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#33

Post by beardicus »

reubano wrote: post #17
beardicus wrote: post #16

Earlier quoted context omitted.

Node-RED is pretty great. I think a wiring interface is a better choice than a block-level thing like blockly.

True, Node-RED is much more aligned to the original Yahoo! Pipes interface. I kinda like the scratch/blockly interface since you can make it clear which components can go where. Plus it's reminiscent of physical Lego blocks. Which interface do you think is more newbie friendly? My gut says blocks (maybe something a bit more simple/refined than blocky) are easier to grok, while wires allow for designing more complex w…

I don't feel like blocks really maps well to the kinds of tasks i'd do in node-red and yahoo pipes, and presumably riko (I didn't dive too deep here yet though). a wiring interface better reflects the idea of inputs, filters, and outputs, and wiring them up in flexible ways. blocks seems a little rigid and more variable/iteration/function-based.

maybe figure out a few common workflows that people would make in riko or node-red, and mock up how they'd work/look in blocks vs. wiring/pipes.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#34

Post by alexchamberlain »

reubano wrote: post #2

`riko` is pure python stream processing library for analyzing and processing streams of structured data. It's modeled after Yahoo! Pipes [1] and was originally a fork of pipe2py [2]. It has both synchronous and asynchronous (via twisted) APIs, and supports parallel execution (via multiprocessing). Out of the box, `riko` can read csv/xml/json/html files; create text and data based flows via modular pipes; parse and ex…

This is an implementation of "Flow Based Programming", right? Its a programming paradigm invented before its time IMHO; perfect for a world of streaming data.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#35

Post by rubber_duck »

olviko wrote: post #31
rubber_duck wrote: post #27

Earlier quoted context omitted.

Maybe I'm not seeing something but wouldn't this be simpler with http://reactivex.io/ (Rx.NET specifically) ? Just implement IObservable that pushes RSS events and then use Rx Filter/Select/Merge/GroupBy to filter/join/synchronize/whatever

Maybe. Don't think any of this existed when I need it... 1. The code is tiny with 90% of it dealing with RSS parsing and filtering. Using RX.NET wouldn't really simplify anything. 2. I wanted a library that I can integrate into my apps and run locally to avoid throttling, robots.txt and other BS Yahoo Pipes was suffering from. I am also not a huge fan of RX... to put it mildly

I think I've used Rx way back when in 2009 (there was no TPL in silverlight so you had to use callbacks and events for continuations which was insane, so I used Rx) so it's been around.

Mind sharing why you don't like it ?

I personally like how it allows me to express complex high level operations cleanly. For example - I have a observable configuration variable that can come from different sources and the source change dynamically. I need to listen to latest source until a new one becomes active - in Rx I only need to push the new source trough IObservable> and then use http://reactivex.io/documentation/operators/switch.html which returns IObservable which will push values from the latest source - Rx will handle unsubscribing from previous active source, synchronizing state and making sure everything is thread safe. And there are a bunch of operators like this that would be tedious and hard implement correctly with all the edge cases in a thread safe way - and here they are abstracted in to high level operators.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#36

Post by reubano »

alexchamberlain wrote: post #34
reubano wrote: post #2

`riko` is pure python stream processing library for analyzing and processing streams of structured data. It's modeled after Yahoo! Pipes [1] and was originally a fork of pipe2py [2]. It has both synchronous and asynchronous (via twisted) APIs, and supports parallel execution (via multiprocessing). Out of the box, `riko` can read csv/xml/json/html files; create text and data based flows via modular pipes; parse and ex…

This is an implementation of "Flow Based Programming", right? Its a programming paradigm invented before its time IMHO; perfect for a world of streaming data.

> This is an implementation of "Flow Based Programming", right?

pretty much.... just without a GUI. My inspiration was method chaining [1] and the first implementation was this [2].

[1] http://martinfowler.com/articles/collection-pipeline/

[2] http://stackoverflow.com/questions/12172934/method-chaining-...

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#38

Post by reubano »

pastaking wrote: post #37

Also might want to check out http://concord.io , it's a bit more work to set up, but it's much faster than most stream processing systems

How does concord differ from the others? spark/storm/flink/etc...? Aside from being written in C that is.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#39

Post by greenshackle »

olviko wrote: post #28
reubano wrote: post #22

Earlier quoted context omitted.

Cool! I actually starred your project last year. Never really got around to looking under the hood though. How do you handle the "multiple destinations" part? In python you can do it with a coroutines [1, 2] implementation (push based). I avoided that, since it coroutines have their own form of callback hell, and decided to implement a generator api (pull based) [3, 4]. But since generators can only be "pulled" into…

Can’t really remember, sorry, that code was built for processing RSS feeds and data size was never an issue. I will take a look when get some free time… But, I see what you mean. I had to deal with similar issues in commercial projects and the "pull" model (generators in Python ~ "yield return" in C#) almost never a good idea, especially when you have to have concurrent consumers. While callbacks are hard to combine,…

Python 3.5 introduced the async/await syntax, I don't do C# but at a glance it's the same.

I've been working on a similar project and I've also found the push model easier.

Post reply on HN