Onyx: fault tolerant data processing for Clojure

21–22 of 22 posts

Re: Onyx: fault tolerant data processing for Clojure

#21

Post by jwr » Fri, Mar 06, 2015, 6:27 PM UTC

XPherior wrote: post #8

jeletonskelly wrote: post #5

Earlier quoted context omitted.

I'm interested to know if you've used Storm at all and how it compares to Onyx. I'm currently considering both for a project.

Hello, Michael Drogalis - the author here. I'm also not a Spark user, but I have used Storm: - Storm is significantly more mature and performant the moment. - Storm has a better cross-language story in terms of bolt functions. - Pretty much everything in Onyx is much more open ended. This applies to deployment, program structure, and workflow creation - and is mostly an artifact of how aggressively Onyx uses data str…

As someone who has been using Storm, this looks very interesting. What I particularly like are the clean, well thought-out ideas. Also, easily reconfigurable (at runtime) topologies are something we'd be interested in. I will definitely take a very close look at Onyx.

Performance is important: in our case, decreasing it significantly below Storm's level would not be acceptable.

Also, I watched the Strange Loop presentation and the tree model looks limiting to me: I have topologies where I need to merge information from two streams (but perhaps I haven't understood the Onyx model yet).

Re: Onyx: fault tolerant data processing for Clojure

#22

Post by XPherior » Fri, Mar 06, 2015, 6:36 PM UTC

jwr wrote: post #21

XPherior wrote: post #8

Earlier quoted context omitted.

Hello, Michael Drogalis - the author here. I'm also not a Spark user, but I have used Storm: - Storm is significantly more mature and performant the moment. - Storm has a better cross-language story in terms of bolt functions. - Pretty much everything in Onyx is much more open ended. This applies to deployment, program structure, and workflow creation - and is mostly an artifact of how aggressively Onyx uses data str…

As someone who has been using Storm, this looks very interesting. What I particularly like are the clean, well thought-out ideas. Also, easily reconfigurable (at runtime) topologies are something we'd be interested in. I will definitely take a very close look at Onyx. Performance is important: in our case, decreasing it significantly below Storm's level would not be acceptable. Also, I watched the Strange Loop presen…

Performance - wait until the 0.6.0 release. We'll be caught up with Storm by then.

The tree model is being removed in 0.6.0 in favor of a vector of vectors (DAG), which allows multiple inputs. See https://github.com/MichaelDrogalis/onyx/blob/0.5.x/doc/user-... The tree model wasn't one of my better ideas.

Edit: to be clear, you can do stream joins right now in 0.5.3 with the DAG model.

Post reply on HN