51–60 of 69 posts
I was a heavy user of pipes and I'm now a heavy user of python. I have built my own dodgy simple replacement for some of the things I used to rely on pipes for. I'm very eager to see what you've got here, at first glance it seems like an excellent fit for my needs. Thanks!
Please let me know what you think. I worked pretty hard on the readme so let me know if anything is confusing and/or doesn't make sense.
Sweet. I put together something similar for NodeJS which is now called 'turtle' (because it's turtles all the way down...). There's a bit of a focus on AWS Lambda & other FaaS solutions as a means of building Lambda architectures, but it can be used by itself. https://github.com/iopipe/turtle
As for riko more specifically, Beam will have soon a python sdk, but I'm unsure if there will be a python standalone runner. Maybe this is something to look into...
[1] https://www.oreilly.com/ideas/future-proof-and-scale-proof-y...
Looks interesting. What kind of applications do people use this for?
Mashups [1] and Extract Transform Load (ETL) [2] are two big use cases. I developed a freelance project aggregator using an earlier version of riko [3]. [1] http://mashable.com/2009/10/08/top-mashups/#0XwtqVCCXPq2 [2] https://www.quora.com/How-do-ETL-tools-work [3] http://app.kazeeki.com/
Earlier quoted context omitted.
Have you investigated any of the existing GUIs? [1-3] I'd love to hear your thoughts on their pros/cons. I do plan to integrate a nice GUI framework if I can find one. [1] https://azkaban.github.io/ [2] https://developers.google.com/blockly/ [3] http://nodered.org/
Node-RED is pretty great. I think a wiring interface is a better choice than a block-level thing like blockly.
`riko` is pure python stream processing library for analyzing and processing streams of structured data. It's modeled after Yahoo! Pipes [1] and was originally a fork of pipe2py [2]. It has both synchronous and asynchronous (via twisted) APIs, and supports parallel execution (via multiprocessing). Out of the box, `riko` can read csv/xml/json/html files; create text and data based flows via modular pipes; parse and ex…
This is an implementation of "Flow Based Programming", right? Its a programming paradigm invented before its time IMHO; perfect for a world of streaming data.
A more modest implementation would be Unix pipes, I think, where the data flows untyped.
Earlier quoted context omitted.
Nice project. I wrote something similar in C# long time ago [1]. Mostly to monitor job feeds and craigslist [2] :-) It supports RSS and Atom, async, various filters, deduplication, etc Yahoo Pipes was a nice project, but as its popularity grew, it started getting blocked more and more. It was also hard to build and maintain pipelines with more than a few steps. [1] https://github.com/olviko/RssPercolator [2] https://…
Cool! I actually starred your project last year. Never really got around to looking under the hood though. How do you handle the "multiple destinations" part? In python you can do it with a coroutines [1, 2] implementation (push based). I avoided that, since it coroutines have their own form of callback hell, and decided to implement a generator api (pull based) [3, 4]. But since generators can only be "pulled" into…
Also might want to check out http://concord.io , it's a bit more work to set up, but it's much faster than most stream processing systems
How does concord differ from the others? spark/storm/flink/etc...? Aside from being written in C that is.
Really cool API, you should port this to concord! =)
i'd say major diff is dynamic topology. So during the pipeline execution you can add/remove workers for any stage.
Also each stage/operator can be written in any programming language.
Storm/Flink/SparkStreaming/etc... all have much higher level API's. We built the execution engine first, these great things (DSL, etc) should come soon. For example this API would be easy to support to execute on top (the pipe abstraction that is)
Here is an example of a DSL we prototyped in a couple hours.
Earlier quoted context omitted.
How does concord differ from the others? spark/storm/flink/etc...? Aside from being written in C that is.
eng at concord here. Really cool API, you should port this to concord! =) i'd say major diff is dynamic topology. So during the pipeline execution you can add/remove workers for any stage. Also each stage/operator can be written in any programming language. Storm/Flink/SparkStreaming/etc... all have much higher level API's. We built the execution engine first, these great things (DSL, etc) should come soon. For examp…