Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Outgrowing Apache Storm: why we built in-house distributed stream processing

libra.to

Post reply on HN
Print view View original

11–20 of 27 posts

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#11

Post by meatysnapper »

virtualwhys wrote: post #4
mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Any consideration given to Scala or Clojure, or did raw performance (via Java 8) win over greater abstraction capabilities from the start? IIRC, Twitter's Storm replacement, Heron, is written in C++ -- i.e. they didn't go with Spark/Scala, which, given that Twitter is probably the largest Scala shop in the world, speaks volumes about the volume of data these sytems need handle (read: Spark is far from slow).

If I remember right, the team that wrote Heron was most comfortable with C++, and that was a major reason in choosing it.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#12

Post by pixelmonkey »

mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

How much of your decision to replace Storm stemmed from "this would be a fun technical challenge" that you economically justified by cutting server costs? It's hard to tell from the blog post whether you view SuperChief as an "indictment" of running Storm at scale, or whether you just thought building your own stream processor would be a great way to learn new things and cut server costs at the same time.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#14

Post by jandrewrogers »

jack9 wrote: post #5
mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Do you using batching to reach that scale of throughput? Streams sometimes are pre-aggregated data and it wasn't clear on if you maintained the granularity through the changes.

I can't speak for their implementation but batching is not necessary. Stream processing complex JSON documents and storing the documents to disk at rates of 500k documents/second per server is demonstrably achievable on some scale-out systems.

The internal architectures make an enormous difference in throughput. A proper high-performance stream processing engine does not look anything like the "Hadoop in RAM" style model.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#16

Post by rayscondition »

doug1001 wrote: post #6
mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

first, thanks for the post--a gold mine of distributed stream-processing knowledge of the type it's probably only possible to acquire the hard way. so obviously Librato develops on the JVM, if you were to begin SuperChief today, now that Akka Steams is 1.0, would you have considered using it? Also, Apache Storm is true one-at-a-time streaming; is SuperChief same or micro-batch? And finally, did i read correctly that…

Thanks! SuperChief does one at a time streaming. The library for leader election is just a wrapper around an apache curator leader latch with some instrumentation and logging so we can reuse it in other services.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#17

Post by alainv »

mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Looks very interesting! Do you plan to open-source SuperChief?

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#18

Post by mike_heffner »

pixelmonkey wrote: post #12
mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

How much of your decision to replace Storm stemmed from "this would be a fun technical challenge" that you economically justified by cutting server costs? It's hard to tell from the blog post whether you view SuperChief as an "indictment" of running Storm at scale, or whether you just thought building your own stream processor would be a great way to learn new things and cut server costs at the same time.

This was not "indictment" of Storm. We made the analysis and determined that in order to achieve the goals of the business we were not going to be able to take our current system to a 10x scale without a significant investment. Like any complex, distributed computing system there is always the question of when it makes sense to replace the portions of the system that you are not able to reason about sufficiently enough to scale.

Superchief is not an off-the-shelf-framework that anyone will be able to drop-in to fulfill their stream processing requirements. We were running on Storm for 2.5 years prior to moving to SC and during that time we truly developed the understanding of the system and our own workload to be able to build SC. SC builds on many assumptions and understandings of our workload that we did not have when we started with Storm.

For us we decided this was the path that would lead to the quickest and highest reward, and we were able to justify the effort by not only halving our infrastructure footprint, but being able to scale the next 10x. Other companies, eg. Twitter Heron, have made similar realizations.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#19

Post by jack9 »

jandrewrogers wrote: post #14
jack9 wrote: post #5

Earlier quoted context omitted.

Do you using batching to reach that scale of throughput? Streams sometimes are pre-aggregated data and it wasn't clear on if you maintained the granularity through the changes.

I can't speak for their implementation but batching is not necessary. Stream processing complex JSON documents and storing the documents to disk at rates of 500k documents/second per server is demonstrably achievable on some scale-out systems. The internal architectures make an enormous difference in throughput. A proper high-performance stream processing engine does not look anything like the "Hadoop in RAM" style m…

> Stream processing complex JSON documents and storing the documents to disk at rates of 500k documents/second per server is demonstrably achievable on some scale-out systems

So is it per server or scaled out? I thought SSDs have capped around 100k discrete per second (P/E aka write cycles).

Can you give an example? I've been unable to practically reach more than a scale of 10k/sec/server using a number of technologies and combinations to collect from socket, parse json and write to socket. That's just my specific use case.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#20

Post by mike_heffner »

alainv wrote: post #17
mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Looks very interesting! Do you plan to open-source SuperChief?

Yes, we do plan to at some point. We want to get a longer production confidence before putting it out there for others and to build necessary documentation that'll make it useful for other teams.
Post reply on HN