Outgrowing Apache Storm: why we built in-house distributed stream processing

#1

Post by librato » Thu, Oct 01, 2015, 6:16 PM UTC

Outgrowing Apache Storm: why we built in-house distributed stream processing
libra.to

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#2

Post by mike_heffner » Thu, Oct 01, 2015, 7:00 PM UTC

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#3

Post by swah » Thu, Oct 01, 2015, 7:16 PM UTC

Also, this book is a great read: https://www.manning.com/books/big-data

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#4

Post by virtualwhys » Thu, Oct 01, 2015, 7:17 PM UTC

mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Any consideration given to Scala or Clojure, or did raw performance (via Java 8) win over greater abstraction capabilities from the start?

IIRC, Twitter's Storm replacement, Heron, is written in C++ -- i.e. they didn't go with Spark/Scala, which, given that Twitter is probably the largest Scala shop in the world, speaks volumes about the volume of data these sytems need handle (read: Spark is far from slow).

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#5

Post by jack9 » Thu, Oct 01, 2015, 7:18 PM UTC

mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Do you using batching to reach that scale of throughput? Streams sometimes are pre-aggregated data and it wasn't clear on if you maintained the granularity through the changes.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#6

Post by doug1001 » Thu, Oct 01, 2015, 7:25 PM UTC

mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

first, thanks for the post--a gold mine of distributed stream-processing knowledge of the type it's probably only possible to acquire the hard way.

so obviously Librato develops on the JVM, if you were to begin SuperChief today, now that Akka Steams is 1.0, would you have considered using it? Also, Apache Storm is true one-at-a-time streaming; is SuperChief same or micro-batch? And finally, did i read correctly that you are using Zookeeper but a separate library for leader election? Does this work w/ the z-nodes or in place of?

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#7

Post by Joeri » Thu, Oct 01, 2015, 7:33 PM UTC

mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

As someone about to invest big time in spark streaming for processing event series data I'm wondering if you could comment on what makes it a bad for for your workload.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#8

Post by henridf » Thu, Oct 01, 2015, 7:33 PM UTC

mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Rather than build your own, did you consider moving to another distributed stream processing platform such as Spark Streaming or Flink? If so, curious to know what disqualified them.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#9

Post by rayscondition » Thu, Oct 01, 2015, 7:33 PM UTC

virtualwhys wrote: post #4

mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Any consideration given to Scala or Clojure, or did raw performance (via Java 8) win over greater abstraction capabilities from the start? IIRC, Twitter's Storm replacement, Heron, is written in C++ -- i.e. they didn't go with Spark/Scala, which, given that Twitter is probably the largest Scala shop in the world, speaks volumes about the volume of data these sytems need handle (read: Spark is far from slow).

Hi, I'm the author of the blog post and wrote SuperChief. We chose Java 8 because it's the language we're most comfortable with on the data team and find it easiest to reason about. We also reused some of the existing bolt code we wrote for Storm to run the time series aggregations. It was already written in Java so we just had to make it thread safe. I suspect you could get similar performance with other jvm based languages.

I was interested in looking at Spark for this for a bit but coming from Storm we decided we wanted something stripped down that was more purpose built.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#10

Post by rayscondition » Thu, Oct 01, 2015, 7:36 PM UTC

jack9 wrote: post #5

mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Do you using batching to reach that scale of throughput? Streams sometimes are pre-aggregated data and it wasn't clear on if you maintained the granularity through the changes.

There's no batching, we have a 1 to 1 mapping of kafka messages to measurements we receive from our api, that could change though over time. Superchief just reads the messages and each message is passed off to another thread for processing.