History of Apache Storm and Lessons Learned

Post by dimon » Tue, Oct 07, 2014, 8:27 AM UTC

discardorama wrote: post #2

Great read. An aside: when Nathan started work on Storm, there was a competing streaming platform being worked on inside Yahoo, called "S4". It had many similarities to Storm, and some key differences. It is still in the incubator: http://incubator.apache.org/s4/ (I have no connection with S4, just know some people who worked on it)

The key advantage Storm brought to the table was, as briefly mentioned in the article, the at-least once processing guarantee Storm offers thanks to its efficient tracking of tuples and their descendants (see https://storm.incubator.apache.org/documentation/Guaranteein...). To my knowledge S4 offers no such guarantees. I think this made Storm more attractive for many use-cases, it did for me.

Post by jtheory » Tue, Oct 07, 2014, 11:57 AM UTC

nemothekid wrote: post #15

jtheory wrote: post #14

I think I get the concept (and it's interesting); but I'm a bit baffled by the terminology: > That led me to the idea of "spouts" and "bolts" – a spout produces brand new streams, and a bolt takes in streams as input and produces streams as output. Spout I can grasp; why "bolt"? Why these two, together? I'd have really appreciated an extra sentence or two here as to why he chose these terms -- naming is so crucial, a…

Bolt here probably references thunderbolt, and they both references (rain)storms.

That was my assumption; but if you're going to name parts of your system after natural phenomena, you have to choose ones that make some sense, not just names out of a hat that are vaguely in the same domain.

"Spout" makes sense as something that's a source of a stream. I haven't been able to figure out the metaphor for "bolt" at all, particularly as something that processes water (stream/spout...) in some way.

Re: History of Apache Storm and Lessons Learned

Re: History of Apache Storm and Lessons Learned