Twitter Heron: Stream Processing at Scale

#1

Post by mekishizufu » Mon, Jun 15, 2015, 11:42 AM UTC

Twitter Heron: Stream Processing at Scale
blog.acolyer.org

Re: Twitter Heron: Stream Processing at Scale

#2

Post by djb_hackernews » Mon, Jun 15, 2015, 4:49 PM UTC

> All the Stream Managers in a topology connect to each other to form an O(k^2) connection network

shouldn't that be !(k-1)? Or am I missing something about the architecture?

Re: Twitter Heron: Stream Processing at Scale

#3

Post by pixelmonkey » Mon, Jun 15, 2015, 4:53 PM UTC

"In the evaluation in section 7 of the paper, Heron beats Storm in every single metric."

In every metric except perhaps the most important one for actual users:

Storm is open source, Heron isn't.

Also, the @TwitterOSS account indicated that there are "no plans to open source in the short-term":

https://twitter.com/TwitterOSS/status/605460461761396736

Re: Twitter Heron: Stream Processing at Scale

#4

Post by frankmcsherry » Mon, Jun 15, 2015, 5:08 PM UTC

pixelmonkey wrote: post #3

"In the evaluation in section 7 of the paper, Heron beats Storm in every single metric." In every metric except perhaps the most important one for actual users: Storm is open source, Heron isn't. Also, the @TwitterOSS account indicated that there are "no plans to open source in the short-term": https://twitter.com/TwitterOSS/status/605460461761396736

The paper contains the motivation: "For our needs at Twitter, we needed a stream processing platform that was open-source, high-performance, scalable, and was compatible with the current Storm API."

I guess 3/4 isn't bad.

Re: Twitter Heron: Stream Processing at Scale

#5

Post by wheaties » Mon, Jun 15, 2015, 5:33 PM UTC

Now I get it. It's not just "I didn't write this, therefore shit, so I need to rewrite it from scratch." This was a clear explanation in a bite sized chunk which illustrated the reasons for Heron. Thanks.

Re: Twitter Heron: Stream Processing at Scale

#6

Post by [deleted] » Mon, Jun 15, 2015, 5:42 PM UTC

djb_hackernews wrote: post #2

> All the Stream Managers in a topology connect to each other to form an O(k^2) connection network shouldn't that be !(k-1)? Or am I missing something about the architecture?

[deleted]

Re: Twitter Heron: Stream Processing at Scale

#7

Post by JSno » Mon, Jun 15, 2015, 5:44 PM UTC

is Heron written in Clojure too? Maybe lack of good Clj programmers made storm hard to maitain?

Re: Twitter Heron: Stream Processing at Scale

#8

Post by escaped_hn » Mon, Jun 15, 2015, 6:44 PM UTC

djb_hackernews wrote: post #2

> All the Stream Managers in a topology connect to each other to form an O(k^2) connection network shouldn't that be !(k-1)? Or am I missing something about the architecture?

Every stream manager is connected to every other stream manager. k managers = k^2 connections.

Re: Twitter Heron: Stream Processing at Scale

#9

Post by dang » Mon, Jun 15, 2015, 6:45 PM UTC

Is the paper online? If someone can produce a standard URL for it, we'll change the link in this post.

Re: Twitter Heron: Stream Processing at Scale

#10

Post by djb_hackernews » Mon, Jun 15, 2015, 7:03 PM UTC

escaped_hn wrote: post #8

djb_hackernews wrote: post #2

> All the Stream Managers in a topology connect to each other to form an O(k^2) connection network shouldn't that be !(k-1)? Or am I missing something about the architecture?

Every stream manager is connected to every other stream manager. k managers = k^2 connections.

Yea I must be missing something here.

If you have k=2 stream managers you have 1 connection, not 4 . Even if the connections weren't bidirectional you'd have 2 connections.