Open-Sourcing Twitter Heron

I've seen a decent amount of complaining about the difficulty of supporting a kafka infrastructure. I'd be interested in thoughts from people who have used Heron as to how it is running it in production.

Re: Open-Sourcing Twitter Heron

#5

Post by itaifrenkel » Wed, May 25, 2016, 4:48 PM UTC

Does anyone know if Storm ShellBolt compatible with Heron, or is there a better way in Heron for running non-jvm bolts?

Re: Open-Sourcing Twitter Heron

#6

Post by elodina » Wed, May 25, 2016, 5:04 PM UTC

dswalter wrote: post #4

I've seen a decent amount of complaining about the difficulty of supporting a kafka infrastructure. I'd be interested in thoughts from people who have used Heron as to how it is running it in production.

We open sourced Kafka support Spout, Bolt & Example Topology https://github.com/twitter/heron/pull/751

We run Kafka Mesos https://github.com/mesos/kafka allows for as a service implementation for broker clusters for users and apps.

Vagrant for running example https://github.com/elodina/heron/blob/v2/contrib/kafka9/vagr...

Re: Open-Sourcing Twitter Heron

#7

Post by zenlikethat » Wed, May 25, 2016, 5:06 PM UTC

Feel like they should feature the link to the repo much more prominently in the article!

For anyone curious, it's https://github.com/twitter/heron

Re: Open-Sourcing Twitter Heron

#8

Post by ghayes » Wed, May 25, 2016, 5:10 PM UTC

zenlikethat wrote: post #7

Feel like they should feature the link to the repo much more prominently in the article! For anyone curious, it's https://github.com/twitter/heron

And they built a site for the project with a good Getting Started section: http://twitter.github.io/heron/

Getting Started: http://twitter.github.io/heron/docs/getting-started/

Re: Open-Sourcing Twitter Heron

#9

Post by lovelearning » Wed, May 25, 2016, 5:21 PM UTC

They have dropped clojure completely. All the critical path messaging in Storm was done using clojure. Dropped netty too. The actual messaging ("stream manager") seems to be C++. Perhaps that explains the latency and CPU improvements. The architectural changes mentioned account for better cluster utilization, fault tolerance and back pressure implementation, but don't explain why raw streaming performance is so much better in Heron.

Edit: Not so sure what they are using for networking. Seemed like cpp-netlib at first, but I don't think so now.

Re: Open-Sourcing Twitter Heron

#10

Post by abc_lisper » Wed, May 25, 2016, 5:26 PM UTC

Wonder how this compares to Storm 1.0