Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Onyx: fault tolerant data processing for Clojure

github.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Onyx: fault tolerant data processing for Clojure

#11

Post by boothead »

erichmond wrote: post #10
boothead wrote: post #7

Looks superficially simmilar to https://github.com/aphyr/tesser anyone know both and can give a comparison? From a brief examination tesser looks a lot simpler (probably because of encoding most of the folding using various monoids). Does onyx have a similar abstraction model that I missed?

Onyx is distributed and Tesser just uses all the available cores of a particular machine AFAIK. Both libraries are awesome.

Tesser also allows you to distribute it using hadoop i think. I haven't used it, I only happened to hear about it why @aphyr gave a talk at the clojure exchange in London.

Re: Onyx: fault tolerant data processing for Clojure

#13

Post by XPherior »

bmh100 wrote: post #12
XPherior wrote: post #9

Hi folks! I'm Michael Drogalis - the primary author. I'm happy to answer any questions.

What were the main pain points that motivated you to develop Onyx? What capabilities do you want to add or have already added that Storm doesn't provide?

See: https://github.com/MichaelDrogalis/onyx/blob/0.5.x/doc/user-...

These are all the things I wrote down that I wanted before I wrote the first line of code.

Re: Onyx: fault tolerant data processing for Clojure

#15

Post by vosper »

XPherior wrote: post #8
jeletonskelly wrote: post #5

Earlier quoted context omitted.

I'm interested to know if you've used Storm at all and how it compares to Onyx. I'm currently considering both for a project.

Hello, Michael Drogalis - the author here. I'm also not a Spark user, but I have used Storm: - Storm is significantly more mature and performant the moment. - Storm has a better cross-language story in terms of bolt functions. - Pretty much everything in Onyx is much more open ended. This applies to deployment, program structure, and workflow creation - and is mostly an artifact of how aggressively Onyx uses data str…

Hi Michael, thanks for your work creating Onyx - it looks really cool.

I can infer two of your frustrations with Storm from the above post: that Storm was too closed, and it's information model didn't span across languages very well. If you have the time, could you elaborate on these pain points, and any others that you found?

Re: Onyx: fault tolerant data processing for Clojure

#16

Post by XPherior »

vosper wrote: post #15
XPherior wrote: post #8

Earlier quoted context omitted.

Hello, Michael Drogalis - the author here. I'm also not a Spark user, but I have used Storm: - Storm is significantly more mature and performant the moment. - Storm has a better cross-language story in terms of bolt functions. - Pretty much everything in Onyx is much more open ended. This applies to deployment, program structure, and workflow creation - and is mostly an artifact of how aggressively Onyx uses data str…

Hi Michael, thanks for your work creating Onyx - it looks really cool. I can infer two of your frustrations with Storm from the above post: that Storm was too closed, and it's information model didn't span across languages very well. If you have the time, could you elaborate on these pain points, and any others that you found?

I'll paraphrase a few snippets from my own documentation to answer these questions. Happy to comment more if needed.

Information models are often superior to APIs, and almost always better than DSLs. The hyper-flexibility of a data structure literal allows Onyx workflows and catalogs to be constructed at a distance, meaning on another machine, in a different language, by another program, etc. Contrast this to Storm. Topologies are written with functions, macros, and objects. These things are specific to a programming language, and make it hard to work at a distance - specifically in the browser. JavaScript is the ultimate place to be when creating specifications.

Further, the information model for an Onyx workflow has the distinct advantage that it's possible to compile other workflows (perhaps a datalog) into the workflow that Onyx understands.

See https://github.com/MichaelDrogalis/onyx/blob/0.5.x/doc/user-... for a continued explanation of why Onyx is more of an "open" concept.

Re: Onyx: fault tolerant data processing for Clojure

#17

Post by pixelmonkey »

XPherior wrote: post #16
vosper wrote: post #15

Earlier quoted context omitted.

Hi Michael, thanks for your work creating Onyx - it looks really cool. I can infer two of your frustrations with Storm from the above post: that Storm was too closed, and it's information model didn't span across languages very well. If you have the time, could you elaborate on these pain points, and any others that you found?

I'll paraphrase a few snippets from my own documentation to answer these questions. Happy to comment more if needed. Information models are often superior to APIs, and almost always better than DSLs. The hyper-flexibility of a data structure literal allows Onyx workflows and catalogs to be constructed at a distance, meaning on another machine, in a different language, by another program, etc. Contrast this to Storm.…

Michael, can you explain this more? "[Storm] Topologies are written with functions, macros, and objects. These things are specific to a programming language, and make it hard to work at a distance -- specifically in the browser. JavaScript is the ultimate place to be when creating specifications."

I don't really get it. Storm Topologies are built in Java or Clojure using a builder interface, but the data structures for topologies themselves are actually DAGs that serialize using Thrift. It's true that this is a bit heavy-weight compared to something like JSON or EDN, but offering an alternative is a discussion in the community right now. What would your ideal representation of topologies be, actually?

Re: Onyx: fault tolerant data processing for Clojure

#18

Post by dj-wonk »

Re: Onyx's architecture. I would wonder about performance when keeping a shared log in ZooKeeper. Why not use something like Kafka -- it is designed for high-volume, immutable logging. ZK works best for less-frequently changing configuration, such as node connection information or snapshotting. I could be wrong. I'd like to hear your thoughts and experience.

Re: Onyx: fault tolerant data processing for Clojure

#19

Post by XPherior »

dj-wonk wrote: post #18

Re: Onyx's architecture. I would wonder about performance when keeping a shared log in ZooKeeper. Why not use something like Kafka -- it is designed for high-volume, immutable logging. ZK works best for less-frequently changing configuration, such as node connection information or snapshotting. I could be wrong. I'd like to hear your thoughts and experience.

- Picking up Kafka means introducing another dependency.

- Onyx's log doesn't grow particularly large because it's only used for coordination, not for messaging.

- Because the log isn't huge, and can be GC'ed, consumers don't experience high volumes of messages.

- ZooKeeper offers sequential node creation - making it a really good fit for what the log needs to do.

Re: Onyx: fault tolerant data processing for Clojure

#20

Post by XPherior »

pixelmonkey wrote: post #17
XPherior wrote: post #16

Earlier quoted context omitted.

I'll paraphrase a few snippets from my own documentation to answer these questions. Happy to comment more if needed. Information models are often superior to APIs, and almost always better than DSLs. The hyper-flexibility of a data structure literal allows Onyx workflows and catalogs to be constructed at a distance, meaning on another machine, in a different language, by another program, etc. Contrast this to Storm.…

Michael, can you explain this more? "[Storm] Topologies are written with functions, macros, and objects. These things are specific to a programming language, and make it hard to work at a distance -- specifically in the browser. JavaScript is the ultimate place to be when creating specifications." I don't really get it. Storm Topologies are built in Java or Clojure using a builder interface, but the data structures f…

I wasn't aware that they're Thrift serializable - that's cool, and offers roughly what Onyx does in terms of its workflow representation.

Onyx goes a little further though in terms of its catalog. I wanted more of the computation to be pulled out into a data structure. That includes runtime parameters, flow, performance tuning knobs, and grouping functions. All of these things are represented as data in Onyx. It's a little harder, at least in my experience, to do these things in Storm.

Post reply on HN