Onyx: fault tolerant data processing for Clojure
1–10 of 22 posts
Re: Onyx: fault tolerant data processing for Clojure
#2This looks very interesting. I'm doing some log file processing in Apache Spark in Clojure. Spark is written in Scala, but has a Java API, which is wrapped by Flambo. It looks and feels entirely Clojure.
The semantics look very similar indeed. Does anyone have a comparison between Onyx and Spark?
Re: Onyx: fault tolerant data processing for Clojure
#3This looks very interesting. I'm doing some log file processing in Apache Spark in Clojure. Spark is written in Scala, but has a Java API, which is wrapped by Flambo. It looks and feels entirely Clojure. The semantics look very similar indeed. Does anyone have a comparison between Onyx and Spark?
I've used Onyx, but I haven't used Spark, so take this with a grain of salt.
A few key differences:
Onyx aggressively uses data structures to define the structure of computation, defining the data flow (Onyx workflow) and parameterization (Onyx catalog) of the the computation via clojure maps and vectors. In comparison Flambo and Spark define the structure of computation via functions over collections. One way in which Onyx's approach is powerful is that it becomes trivial to manipulate workflows or catalogs before submitting jobs at runtime, allowing you to add additional tasks, task options, etc.
Onyx also implements batching over streaming operations, whereas Spark appears to be the opposite. There are likely to be trade-offs between these approaches.
Spark is also a lot faster, though this isn't necessarily intrinsic to the approaches.
Re: Onyx: fault tolerant data processing for Clojure
#4Checkout the original video introducing Onyx: http://youtu.be/vG47Gui3hYE
Re: Onyx: fault tolerant data processing for Clojure
#5This looks very interesting. I'm doing some log file processing in Apache Spark in Clojure. Spark is written in Scala, but has a Java API, which is wrapped by Flambo. It looks and feels entirely Clojure. The semantics look very similar indeed. Does anyone have a comparison between Onyx and Spark?
I've used Onyx, but I haven't used Spark, so take this with a grain of salt. A few key differences: Onyx aggressively uses data structures to define the structure of computation, defining the data flow (Onyx workflow) and parameterization (Onyx catalog) of the the computation via clojure maps and vectors. In comparison Flambo and Spark define the structure of computation via functions over collections. One way in whi…
I'm interested to know if you've used Storm at all and how it compares to Onyx. I'm currently considering both for a project.
Re: Onyx: fault tolerant data processing for Clojure
#6Checkout the original video introducing Onyx: http://youtu.be/vG47Gui3hYE
Live open sourcing !
Re: Onyx: fault tolerant data processing for Clojure
#7Looks superficially simmilar to https://github.com/aphyr/tesser anyone know both and can give a comparison?
From a brief examination tesser looks a lot simpler (probably because of encoding most of the folding using various monoids). Does onyx have a similar abstraction model that I missed?
Re: Onyx: fault tolerant data processing for Clojure
#8Earlier quoted context omitted.
I've used Onyx, but I haven't used Spark, so take this with a grain of salt. A few key differences: Onyx aggressively uses data structures to define the structure of computation, defining the data flow (Onyx workflow) and parameterization (Onyx catalog) of the the computation via clojure maps and vectors. In comparison Flambo and Spark define the structure of computation via functions over collections. One way in whi…
I'm interested to know if you've used Storm at all and how it compares to Onyx. I'm currently considering both for a project.
Hello, Michael Drogalis - the author here.
I'm also not a Spark user, but I have used Storm:
- Storm is significantly more mature and performant the moment.
- Storm has a better cross-language story in terms of bolt functions.
- Pretty much everything in Onyx is much more open ended. This applies to deployment, program structure, and workflow creation - and is mostly an artifact of how aggressively Onyx uses data structures.
- Onyx has a far better reach across languages in terms of its information model.
- Onyx will be adopting a tweaked version of Storm's message model next release to get on the same level of performance and reliability. We're dropping the HornetQ dependency.
- Onyx is born out of years of frustration of direct usage of Storm and Hadoop.
Re: Onyx: fault tolerant data processing for Clojure
#9Hi folks! I'm Michael Drogalis - the primary author. I'm happy to answer any questions.
Re: Onyx: fault tolerant data processing for Clojure
#10Looks superficially simmilar to https://github.com/aphyr/tesser anyone know both and can give a comparison? From a brief examination tesser looks a lot simpler (probably because of encoding most of the folding using various monoids). Does onyx have a similar abstraction model that I missed?
Onyx is distributed and Tesser just uses all the available cores of a particular machine AFAIK.
Both libraries are awesome.