Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

github.com

Post reply on HN
Print view View original

61–69 of 69 posts

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#61

Post by bimil »

reubano wrote: post #38
pastaking wrote: post #37

Also might want to check out http://concord.io , it's a bit more work to set up, but it's much faster than most stream processing systems

How does concord differ from the others? spark/storm/flink/etc...? Aside from being written in C that is.

many differences from spark/storm/flink, but the most notable difference of concord is its dynamic topology model - deployment, scaling, and changes to your topologies all can be done in runtime, w/o restart of the full topology, which is required for spark/storm/flink.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#62

Post by reubano »

infinite8s wrote: post #58
reubano wrote: post #22

Earlier quoted context omitted.

Cool! I actually starred your project last year. Never really got around to looking under the hood though. How do you handle the "multiple destinations" part? In python you can do it with a coroutines [1, 2] implementation (push based). I avoided that, since it coroutines have their own form of callback hell, and decided to implement a generator api (pull based) [3, 4]. But since generators can only be "pulled" into…

Another benefit of the push model is that back pressure becomes much easier to implement.

Any code to share? I've yet to see how to properly do push.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#63

Post by reubano »

ecesena wrote: post #54

This is really interesting. Have you looked at Apache Beam? What I think is interesting about Beam -in this specific context- is that it has a standalone runner (java), that similarly as riko let you write pipelines without worrying about a complex setup. But then, if you need to scale your computation, Beam is runner-independent and you can take the same code and run it at scale on a cluster, wether it's spark, flin…

> This is really interesting. Have you looked at Apache Beam?

Just gave it a look. Took a while to find some examples with code, but once I did it made a bit more sense.

> Beam is runner-independent and you can take the same code and run it at scale on a cluster, wether it's spark, flink, or google cloud.

I thought that was pretty cool.

> As for riko more specifically, Beam will have soon a python sdk, but I'm unsure if there will be a python standalone runner. Maybe this is something to look into...

A python standalone runner would be very useful. Otherwise I'm hesitant to go much further since my goal is to have a pure python solution for working with streaming data. Most libraries require installing java and that is what I'd like to avoid.

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#64

Post by reubano »

bimil wrote: post #61
reubano wrote: post #38

Earlier quoted context omitted.

How does concord differ from the others? spark/storm/flink/etc...? Aside from being written in C that is.

many differences from spark/storm/flink, but the most notable difference of concord is its dynamic topology model - deployment, scaling, and changes to your topologies all can be done in runtime, w/o restart of the full topology, which is required for spark/storm/flink.

> ...all can be done in runtime, w/o restart of the full topology...

That's pretty impressive! So something like react/elm hot reloading?

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#65

Post by reubano »

agallego wrote: post #60
agallego wrote: post #59

Earlier quoted context omitted.

eng at concord here. Really cool API, you should port this to concord! =) i'd say major diff is dynamic topology. So during the pipeline execution you can add/remove workers for any stage. Also each stage/operator can be written in any programming language. Storm/Flink/SparkStreaming/etc... all have much higher level API's. We built the execution engine first, these great things (DSL, etc) should come soon. For examp…

err. missing link: https://github.com/jjmalina/concord-python-dsl

Pretty neat! I'm guessing that concord isn't limited to just map/reduce... correct? I think building in integrations to other systems in the stream processing ecosystem is key. First on the list is to integrate with popular workflow schedulers so that you can design topologies which riko would then parse.

Next up I think would be supporting custom sources/sinks such as Twitter, HDFS, RDMS, etc. What exactly would be involved in "porting" riko to concord and what would the advantages be for doing so?

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#66

Post by reubano »

beardicus wrote: post #33
reubano wrote: post #17

Earlier quoted context omitted.

True, Node-RED is much more aligned to the original Yahoo! Pipes interface. I kinda like the scratch/blockly interface since you can make it clear which components can go where. Plus it's reminiscent of physical Lego blocks. Which interface do you think is more newbie friendly? My gut says blocks (maybe something a bit more simple/refined than blocky) are easier to grok, while wires allow for designing more complex w…

I don't feel like blocks really maps well to the kinds of tasks i'd do in node-red and yahoo pipes, and presumably riko (I didn't dive too deep here yet though). a wiring interface better reflects the idea of inputs, filters, and outputs, and wiring them up in flexible ways. blocks seems a little rigid and more variable/iteration/function-based. maybe figure out a few common workflows that people would make in riko o…

> maybe figure out a few common workflows that people would make in riko or node-red, and mock up how they'd work/look in blocks vs. wiring/pipes.

Good idea, what are your workflows?

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#67

Post by bimil »

reubano wrote: post #64
bimil wrote: post #61

Earlier quoted context omitted.

many differences from spark/storm/flink, but the most notable difference of concord is its dynamic topology model - deployment, scaling, and changes to your topologies all can be done in runtime, w/o restart of the full topology, which is required for spark/storm/flink.

> ...all can be done in runtime, w/o restart of the full topology... That's pretty impressive! So something like react/elm hot reloading?

Yeah I guess that's one way to put it (little different but kindof).

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#68

Post by agallego »

reubano wrote: post #65
agallego wrote: post #60

Earlier quoted context omitted.

err. missing link: https://github.com/jjmalina/concord-python-dsl

Pretty neat! I'm guessing that concord isn't limited to just map/reduce... correct? I think building in integrations to other systems in the stream processing ecosystem is key. First on the list is to integrate with popular workflow schedulers so that you can design topologies which riko would then parse. Next up I think would be supporting custom sources/sinks such as Twitter, HDFS, RDMS, etc. What exactly would be…

Interesting, will keep an eye out as you mature it, looks like an awesome DSL for us to fill the runtime.

Advantages are:

1. mesos integration, with that comes containerization support, multi tenancy, QoS, proper pipeline supervision, etc 2. Scheduling of pipelines. i.e.: Schedule them on 100 computers. 3. the outputs of your DAG could be consumed by other systems immediately and even written in different programming languages. So your python DSL could be the source to the Scala DSL at some point. so language interop 4. Available KV storage 5. Tracing (Zipkin) - ala Google Dapper. 6. Fast networking - C++ backed runtime is 1 order of magnitude faster than the python one.

What would be involved, is not much from what I can tell.

Each concord 'operator' is like a networked function.

so given a DAG, you could generate many operators internally, or literally write them to a file, i.e.: operator_one.py etc. The code generation or internal scheduling would be the glue that's needed.

if you ever become interested, ping me! would love to collab alex@concord.io

Re: Show HN: Riko – A Python stream processing engine modeled after Yahoo! Pipes

#69

Post by ecesena »

reubano wrote: post #63
ecesena wrote: post #54

This is really interesting. Have you looked at Apache Beam? What I think is interesting about Beam -in this specific context- is that it has a standalone runner (java), that similarly as riko let you write pipelines without worrying about a complex setup. But then, if you need to scale your computation, Beam is runner-independent and you can take the same code and run it at scale on a cluster, wether it's spark, flin…

> This is really interesting. Have you looked at Apache Beam? Just gave it a look. Took a while to find some examples with code, but once I did it made a bit more sense. > Beam is runner-independent and you can take the same code and run it at scale on a cluster, wether it's spark, flink, or google cloud. I thought that was pretty cool. > As for riko more specifically, Beam will have soon a python sdk, but I'm unsure…

Python sdk is work in progress - there's currently a branch: https://github.com/apache/incubator-beam/tree/python-sdk
Post reply on HN