Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Outgrowing Apache Storm: why we built in-house distributed stream processing

libra.to

Post reply on HN
Print view View original

21–27 of 27 posts

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#21

Post by mike_heffner »

henridf wrote: post #8
mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

Rather than build your own, did you consider moving to another distributed stream processing platform such as Spark Streaming or Flink? If so, curious to know what disqualified them.

Yes, we did survey the solutions out there at the time. At the end of the day the decision was really made on the fact that we knew the domain and problem we were trying to solve with Storm very well. The same topologies we've replaced with Superchief have been running on Storm for years and the business logic of them has not changed that significantly. Given the team's experience, it was easier for us to replace the streaming plumbing than become experts on another abstraction framework.

So this is definitely not a dig at any framework. If we were starting to build stream processing into our infrastructure today we would most likely start on an existing framework. Also, I imagine as our needs grow we will leverage these frameworks in future proof-of-concept streaming projects that may or may not make sense to move to Superchief.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#22

Post by mike_heffner »

Joeri wrote: post #7
mike_heffner wrote: post #2

I'm on the data team at Librato, happy to answer any questions. Some of the Librato team will also be at ReInvent next week, can discuss more in person for others attending.

As someone about to invest big time in spark streaming for processing event series data I'm wondering if you could comment on what makes it a bad for for your workload.

See other comment about Spark and frameworks. If you are just starting out with stream processing I would highly recommend using the best-of-breed frameworks. Frameworks provide abstractions that let you focus on solving your domain problems quickly, you don't want to have to worry about the plumbing until you've solved those.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#23

Post by lcr »

jack9 wrote: post #19
jandrewrogers wrote: post #14

Earlier quoted context omitted.

I can't speak for their implementation but batching is not necessary. Stream processing complex JSON documents and storing the documents to disk at rates of 500k documents/second per server is demonstrably achievable on some scale-out systems. The internal architectures make an enormous difference in throughput. A proper high-performance stream processing engine does not look anything like the "Hadoop in RAM" style m…

> Stream processing complex JSON documents and storing the documents to disk at rates of 500k documents/second per server is demonstrably achievable on some scale-out systems So is it per server or scaled out? I thought SSDs have capped around 100k discrete per second (P/E aka write cycles). Can you give an example? I've been unable to practically reach more than a scale of 10k/sec/server using a number of technologi…

Looking at the top end of Intel's SSD lineup I see that they have a product that advertises up to 175k IOPS of random 4K writes. Is this what you are referring?

The product is the 2TB P3700.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#24

Post by henridf »

mike_heffner wrote: post #21
henridf wrote: post #8

Earlier quoted context omitted.

Rather than build your own, did you consider moving to another distributed stream processing platform such as Spark Streaming or Flink? If so, curious to know what disqualified them.

Yes, we did survey the solutions out there at the time. At the end of the day the decision was really made on the fact that we knew the domain and problem we were trying to solve with Storm very well. The same topologies we've replaced with Superchief have been running on Storm for years and the business logic of them has not changed that significantly. Given the team's experience, it was easier for us to replace the…

Makes sense. How do you deal with data arriving out-of-order? (assuming your data is timestamped at the source)

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#25

Post by mike_heffner »

henridf wrote: post #24
mike_heffner wrote: post #21

Earlier quoted context omitted.

Yes, we did survey the solutions out there at the time. At the end of the day the decision was really made on the fact that we knew the domain and problem we were trying to solve with Storm very well. The same topologies we've replaced with Superchief have been running on Storm for years and the business logic of them has not changed that significantly. Given the team's experience, it was easier for us to replace the…

Makes sense. How do you deal with data arriving out-of-order? (assuming your data is timestamped at the source)

We grab a timestamp when data hits our API (largely as close to source as we can get) and use that to order time samples. For time series the ordering requirements aren't as strict as other problem domains, since we can usually go back and overwrite previous time series data points.

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#26

Post by pixelmonkey »

mike_heffner wrote: post #18
pixelmonkey wrote: post #12

Earlier quoted context omitted.

How much of your decision to replace Storm stemmed from "this would be a fun technical challenge" that you economically justified by cutting server costs? It's hard to tell from the blog post whether you view SuperChief as an "indictment" of running Storm at scale, or whether you just thought building your own stream processor would be a great way to learn new things and cut server costs at the same time.

This was not "indictment" of Storm. We made the analysis and determined that in order to achieve the goals of the business we were not going to be able to take our current system to a 10x scale without a significant investment. Like any complex, distributed computing system there is always the question of when it makes sense to replace the portions of the system that you are not able to reason about sufficiently enou…

Makes total sense, especially the part "Superchief is not an off-the-shelf-framework that anyone will be able to drop-in to fulfill their stream processing requirements".

So, essentially, Storm solved a more generic problem than you needed it to solve. Superchief is a narrower system that more closely aligns with your problem domain, and this let you not only simplify a little but also gain efficiency. Makes sense!

(p.s. the reason I ask is because I am deeply committed to the Apache Storm open source community, as one of the co-authors of streamparse and pystorm.)

Re: Outgrowing Apache Storm: why we built in-house distributed stream processing

#27

Post by scaleout1 »

meatysnapper wrote: post #11
virtualwhys wrote: post #4

Earlier quoted context omitted.

Any consideration given to Scala or Clojure, or did raw performance (via Java 8) win over greater abstraction capabilities from the start? IIRC, Twitter's Storm replacement, Heron, is written in C++ -- i.e. they didn't go with Spark/Scala, which, given that Twitter is probably the largest Scala shop in the world, speaks volumes about the volume of data these sytems need handle (read: Spark is far from slow).

If I remember right, the team that wrote Heron was most comfortable with C++, and that was a major reason in choosing it.

Heron is half Java and half c++ and could have been written on top of finagle had a different team inside Twitter had implemented it
Post reply on HN