Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

github.com

Post reply on HN
Print view View original

11–20 of 69 posts

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#13

Post by reubano »

tudorw wrote: post #10

if someone can spin up a usable gui, charge enough to make a living without compromising on performance, promise some longevity and a way to export of my stuff I would probably pay for that, I loved pipes, the GUI was a big deal for me.

Have you investigated any of the existing GUIs? [1-3] I'd love to hear your thoughts on their pros/cons. I do plan to integrate a nice GUI framework if I can find one.

[1] https://azkaban.github.io/ [2] https://developers.google.com/blockly/ [3] http://nodered.org/

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#15

Post by reubano »

oellegaard wrote: post #12

If you're looking for a stream processing engine more close to Storm, etc. but also simple, check out Motorway: https://github.com/plecto/motorway :-)

Interesting project. I hadn't come across this one yet. One difference that riko has is it's based around functions whereas this library (and practically every stream processing lib I've come across) is based around classes.

I personally prefer the functional approach much better. And if you compare the word count examples on the respective readmes [1, 2], you will see riko is much more succinct. But I suppose the verbosity of the other libraries come with benefits like scaling across a cluster of servers.

[1] https://github.com/plecto/motorway#word-count-example

[2] https://github.com/nerevu/riko#word-count

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#16

Post by beardicus »

reubano wrote: post #13
tudorw wrote: post #10

if someone can spin up a usable gui, charge enough to make a living without compromising on performance, promise some longevity and a way to export of my stuff I would probably pay for that, I loved pipes, the GUI was a big deal for me.

Have you investigated any of the existing GUIs? [1-3] I'd love to hear your thoughts on their pros/cons. I do plan to integrate a nice GUI framework if I can find one. [1] https://azkaban.github.io/ [2] https://developers.google.com/blockly/ [3] http://nodered.org/

Node-RED is pretty great. I think a wiring interface is a better choice than a block-level thing like blockly.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#17

Post by reubano »

beardicus wrote: post #16
reubano wrote: post #13

Earlier quoted context omitted.

Have you investigated any of the existing GUIs? [1-3] I'd love to hear your thoughts on their pros/cons. I do plan to integrate a nice GUI framework if I can find one. [1] https://azkaban.github.io/ [2] https://developers.google.com/blockly/ [3] http://nodered.org/

Node-RED is pretty great. I think a wiring interface is a better choice than a block-level thing like blockly.

True, Node-RED is much more aligned to the original Yahoo! Pipes interface. I kinda like the scratch/blockly interface since you can make it clear which components can go where. Plus it's reminiscent of physical Lego blocks.

Which interface do you think is more newbie friendly? My gut says blocks (maybe something a bit more simple/refined than blocky) are easier to grok, while wires allow for designing more complex workflows.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#19

Post by reubano »

tanlermin wrote: post #18

Can you consider Dask integration? http://distributed.readthedocs.io/en/latest/queues.html https://github.com/dask/dask It can handle parallel and distributed parts for you. https://github.com/dask/dask

I just read about dask earlier today, very neat project! riko already handles parallel processing [1] but adding distributed processing sounds tempting. TBH though, distribution isn't high on the priority list. But I'll be happy to accept a PR if you are so inclined :)

[1] https://github.com/nerevu/riko#parallel-processing

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#20

Post by olviko »

reubano wrote: post #2

`riko` is pure python stream processing library for analyzing and processing streams of structured data. It's modeled after Yahoo! Pipes [1] and was originally a fork of pipe2py [2]. It has both synchronous and asynchronous (via twisted) APIs, and supports parallel execution (via multiprocessing). Out of the box, `riko` can read csv/xml/json/html files; create text and data based flows via modular pipes; parse and ex…

Nice project. I wrote something similar in C# long time ago [1]. Mostly to monitor job feeds and craigslist [2] :-) It supports RSS and Atom, async, various filters, deduplication, etc

Yahoo Pipes was a nice project, but as its popularity grew, it started getting blocked more and more. It was also hard to build and maintain pipelines with more than a few steps.

[1] https://github.com/olviko/RssPercolator

[2] https://github.com/olviko/RssPercolator/blob/master/RssPerco...

Post reply on HN