Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

github.com

Post reply on HN
Print view View original

1–10 of 69 posts

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#2

Post by reubano »

`riko` is pure python stream processing library for analyzing and processing streams of structured data. It's modeled after Yahoo! Pipes [1] and was originally a fork of pipe2py [2]. It has both synchronous and asynchronous (via twisted) APIs, and supports parallel execution (via multiprocessing).

Out of the box, `riko` can read csv/xml/json/html files; create text and data based flows via modular pipes; parse and extract RSS/ATOM feeds; and bunch of other neat things. You can think of `riko` as a poor man's Spark/Storm... stream processing made easy!

Feedback welcome so let me know what you think!

Resources: FAQ [3], cookbook [4], and ipython notebook [5]

Quickie Demo:

    >>> from riko.modules import fetch
    >>> 
    >>> stream = fetch.pipe(conf={'url': 'https://news.ycombinator.com/rss'})
    >>> item = next(stream)
    >>> item['title'], item['link']
    ('Master Plan, Part Deux', 'https://www.tesla.com/blog/master-plan-part-deux')
[1] https://web.archive.org/web/20150930021241/http://pipes.yaho...

[2] https://github.com/ggaughan/pipe2py/

[3] https://github.com/nerevu/riko/blob/master/docs/FAQ.rst

[4] https://github.com/nerevu/riko/blob/master/docs/COOKBOOK.rst

[5] http://nbviewer.jupyter.org/github/nerevu/riko/blob/master/e...

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#3

Post by Fuzzwah »

I was a heavy user of pipes and I'm now a heavy user of python. I have built my own dodgy simple replacement for some of the things I used to rely on pipes for. I'm very eager to see what you've got here, at first glance it seems like an excellent fit for my needs.

Thanks!

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#4

Post by reubano »

Fuzzwah wrote: post #3

I was a heavy user of pipes and I'm now a heavy user of python. I have built my own dodgy simple replacement for some of the things I used to rely on pipes for. I'm very eager to see what you've got here, at first glance it seems like an excellent fit for my needs. Thanks!

Please let me know what you think. I worked pretty hard on the readme so let me know if anything is confusing and/or doesn't make sense.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#8

Post by reubano »

satai wrote: post #5

Looks nice. Are there any plans for twitter support?

Eventually. It would essentially be a "source" pipe. But ultimately, I want to build a plugin system so that end users can create/share their own pipes. I also plan to add pipes that let you add streams to a database.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#9

Post by reubano »

raimue wrote: post #6

I am still a user of Plagger [1], but development halted quite some time ago. Maybe this could be a good replacement. [1] https://github.com/miyagawa/plagger

Hadn't heard of this one before. There isn't a readme and the site seems to have been taken over. But from what I can tell based on the examples, it has a yaml based scheduler and does some pretty nifty things like IRC notifications.

riko doesn't have a scheduler (although the original pipe2py has a json based one). However, I do plan to integrate with Airflow/Oozie/Luigi [1-3] in the future to make it easier to design workflows.

The notification system reminds me of Huggin [4]. Since riko is twisted based, it should be fairly straightforward to implement something similar for IRC/IMAP/FTP/etc.

[1] https://github.com/apache/incubator-airflow

[2] http://oozie.apache.org/

[3] https://github.com/spotify/luigi

[4] https://github.com/cantino/huginn

Post reply on HN