Looks interesting. What kind of applications do people use this for?
Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
21–30 of 69 posts
Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
#22`riko` is pure python stream processing library for analyzing and processing streams of structured data. It's modeled after Yahoo! Pipes [1] and was originally a fork of pipe2py [2]. It has both synchronous and asynchronous (via twisted) APIs, and supports parallel execution (via multiprocessing). Out of the box, `riko` can read csv/xml/json/html files; create text and data based flows via modular pipes; parse and ex…
Nice project. I wrote something similar in C# long time ago [1]. Mostly to monitor job feeds and craigslist [2] :-) It supports RSS and Atom, async, various filters, deduplication, etc Yahoo Pipes was a nice project, but as its popularity grew, it started getting blocked more and more. It was also hard to build and maintain pipelines with more than a few steps. [1] https://github.com/olviko/RssPercolator [2] https://…
Cool! I actually starred your project last year. Never really got around to looking under the hood though. How do you handle the "multiple destinations" part? In python you can do it with a coroutines [1, 2] implementation (push based). I avoided that, since it coroutines have their own form of callback hell, and decided to implement a generator api (pull based) [3, 4].
But since generators can only be "pulled" into one destination, you have to copy a stream (subsequently converting it into a list) if you want more than one destination [5]. This works fine if the data can fit in memory, but if it can't then you're out of luck!
[1] http://www.dabeaz.com/coroutines/copipe.py
[2] http://www.dabeaz.com/coroutines/
[3] http://www.dabeaz.com/generators/retuple.py
[4] http://www.dabeaz.com/generators
[5] https://github.com/nerevu/riko/blob/master/riko/modules/spli...
Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
#23Also in this space (and worth looking at for inspiration, especially for other potential sources and sinks of data) - Apache Camel [1].
Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
#24if someone can spin up a usable gui, charge enough to make a living without compromising on performance, promise some longevity and a way to export of my stuff I would probably pay for that, I loved pipes, the GUI was a big deal for me.
Just curious:
What kind of a demand is there for a pipes-kind of product or even a customizable/searchable rss/feed integrator?
How much would a typical user be willing to pay for it?
Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
#25If you're looking for a stream processing engine more close to Storm, etc. but also simple, check out Motorway: https://github.com/plecto/motorway :-)
Interesting project. I hadn't come across this one yet. One difference that riko has is it's based around functions whereas this library (and practically every stream processing lib I've come across) is based around classes. I personally prefer the functional approach much better. And if you compare the word count examples on the respective readmes [1, 2], you will see riko is much more succinct. But I suppose the ve…
I just started writing a functional stream processing library in Python for some of the same reasons.
We use somewhat different concepts. I tend to think of streams as infinite, so it didn't occur to me to include something like a reverse pipe operator.
I'm a bit surprised, why is filter an operator rather than a processor? I would think filters usually apply per-item, not to a whole stream?
I havn't worked on it very much but I'm heading towards push-based, using 0MQ for distribution/parallel processing, and using asyncio, mostly because it plays nicely with 0MQ.
Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
#26Looks interesting. What kind of applications do people use this for?
Mashups [1] and Extract Transform Load (ETL) [2] are two big use cases. I developed a freelance project aggregator using an earlier version of riko [3].
[1] http://mashable.com/2009/10/08/top-mashups/#0XwtqVCCXPq2
Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
#27`riko` is pure python stream processing library for analyzing and processing streams of structured data. It's modeled after Yahoo! Pipes [1] and was originally a fork of pipe2py [2]. It has both synchronous and asynchronous (via twisted) APIs, and supports parallel execution (via multiprocessing). Out of the box, `riko` can read csv/xml/json/html files; create text and data based flows via modular pipes; parse and ex…
Nice project. I wrote something similar in C# long time ago [1]. Mostly to monitor job feeds and craigslist [2] :-) It supports RSS and Atom, async, various filters, deduplication, etc Yahoo Pipes was a nice project, but as its popularity grew, it started getting blocked more and more. It was also hard to build and maintain pipelines with more than a few steps. [1] https://github.com/olviko/RssPercolator [2] https://…
Maybe I'm not seeing something but wouldn't this be simpler with http://reactivex.io/ (Rx.NET specifically) ? Just implement IObservable that pushes RSS events and then use Rx Filter/Select/Merge/GroupBy to filter/join/synchronize/whatever
Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
#28Earlier quoted context omitted.
Nice project. I wrote something similar in C# long time ago [1]. Mostly to monitor job feeds and craigslist [2] :-) It supports RSS and Atom, async, various filters, deduplication, etc Yahoo Pipes was a nice project, but as its popularity grew, it started getting blocked more and more. It was also hard to build and maintain pipelines with more than a few steps. [1] https://github.com/olviko/RssPercolator [2] https://…
Cool! I actually starred your project last year. Never really got around to looking under the hood though. How do you handle the "multiple destinations" part? In python you can do it with a coroutines [1, 2] implementation (push based). I avoided that, since it coroutines have their own form of callback hell, and decided to implement a generator api (pull based) [3, 4]. But since generators can only be "pulled" into…
Can’t really remember, sorry, that code was built for processing RSS feeds and data size was never an issue. I will take a look when get some free time…
But, I see what you mean. I had to deal with similar issues in commercial projects and the "pull" model (generators in Python ~ "yield return" in C#) almost never a good idea, especially when you have to have concurrent consumers. While callbacks are hard to combine, in C# it can be nicely abstracted with “async/await”, not sure how it is handled in Python, I stopped using it around 2.5
Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
#29Is there anything like this available that's based on node.js with a decent GUI?
Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes
#30Earlier quoted context omitted.
Interesting project. I hadn't come across this one yet. One difference that riko has is it's based around functions whereas this library (and practically every stream processing lib I've come across) is based around classes. I personally prefer the functional approach much better. And if you compare the word count examples on the respective readmes [1, 2], you will see riko is much more succinct. But I suppose the ve…
I just started writing a functional stream processing library in Python for some of the same reasons. We use somewhat different concepts. I tend to think of streams as infinite, so it didn't occur to me to include something like a reverse pipe operator. I'm a bit surprised, why is filter an operator rather than a processor? I would think filters usually apply per-item, not to a whole stream? I havn't worked on it ver…
> We use somewhat different concepts. I tend to think of streams as infinite, so it didn't occur to me to include something like a reverse pipe operator.
We are in agreement. reverse has a notice that it isn't lazy [1]. I prefer to include pipes that aren't lazy since it can be helpful in some cases (plus the goal is to include all pipes originally in Yahoo! Pipes). The vast majority of pipes work just fine on infinite streams [2].
> I'm a bit surprised, why is filter an operator rather than a processor? I would think filters usually apply per-item, not to a whole stream?
Just an implementation detail [3]. I agree it would be better if it were a processor since it could be parallelized. PRs welcome :).
> I haven't worked on it very much but I'm heading towards push-based, using 0MQ for distribution/parallel processing, and using asyncio, mostly because it plays nicely with 0MQ.
See my previous comments related to this [4, 5].
[1] https://github.com/nerevu/riko/blob/master/riko/modules/reve...
[2] Assuming you're not using the async or parallel mode
[3] https://github.com/nerevu/riko/blob/master/riko/modules/filt...