History of Apache Storm and Lessons Learned
nathanmarz.com
History of Apache Storm and Lessons Learned
1–10 of 22 posts
Re: History of Apache Storm and Lessons Learned
#2Great read.
An aside: when Nathan started work on Storm, there was a competing streaming platform being worked on inside Yahoo, called "S4". It had many similarities to Storm, and some key differences.
It is still in the incubator: http://incubator.apache.org/s4/ (I have no connection with S4, just know some people who worked on it)
Re: History of Apache Storm and Lessons Learned
#3I've been using Storm for a few years in production. Thanks for all the hard work Nathan.
You and Kyle Kingsbury are two "younger" guys that I always enjoy reading whatever you write.
Re: History of Apache Storm and Lessons Learned
#4I've seen Samza's documentation comparing itself to Storm, I wonder what the reverse-perspective is...
Re: History of Apache Storm and Lessons Learned
#5How does it compare to Apache Kafka? (I haven't used either.)
Re: History of Apache Storm and Lessons Learned
#6How does it compare to Apache Kafka? (I haven't used either.)
Kafka is a high performance, persistent message queue.
Storm is distributed stream processing.
The two work together very well. Often, a server will publish messages to Kafka (or another queue), which are then read and processed by Storm.
Re: History of Apache Storm and Lessons Learned
#7I've seen Samza's documentation comparing itself to Storm, I wonder what the reverse-perspective is...
As I understood it Samza ties in with Hadoop/HBase, so if you already have that, it might be easier (or even possible at all) to integrate, whereas Storm has nothing to do with with that and only does stream processing.
Re: History of Apache Storm and Lessons Learned
#8How does it compare to Apache Kafka? (I haven't used either.)
Kafka is a high performance, persistent message queue. Storm is distributed stream processing. The two work together very well. Often, a server will publish messages to Kafka (or another queue), which are then read and processed by Storm.
Interesting question and response. I haven't used either but read about both systems independently. A bit more googling and I found:
http://www.slideshare.net/gschmutz/kafka-andstromeventproces...
Slides 42 and 43 describe an architecture with all three ... hadoop, storm and kafka. Seems like a beast and I don't have applications but pretty cool combo of technologies.
Re: History of Apache Storm and Lessons Learned
#9Nice read. At my previous job we used storm for all sorts of cool things. It worked great and it was a pleasure to be the one at the company that got to really cozy up with it. Deployment (with or without storm-deploy) was also well thought out.
Re: History of Apache Storm and Lessons Learned
#10Earlier quoted context omitted.
Kafka is a high performance, persistent message queue. Storm is distributed stream processing. The two work together very well. Often, a server will publish messages to Kafka (or another queue), which are then read and processed by Storm.
Interesting question and response. I haven't used either but read about both systems independently. A bit more googling and I found: http://www.slideshare.net/gschmutz/kafka-andstromeventproces... Slides 42 and 43 describe an architecture with all three ... hadoop, storm and kafka. Seems like a beast and I don't have applications but pretty cool combo of technologies.
http://manning.com/marz/ - this is often the trifecta used in the "lambda architecture", coined by nathan marz. This is also the system they built @ backtype to do their real-time analytics system.