Open-Sourcing Twitter Heron • Hacker News Board

Re: Open-Sourcing Twitter Heron

#11

Post by scaleout1 » Wed, May 25, 2016, 5:31 PM UTC

As someone who has used Heron (along with MillWheel, Spark Streaming and Storm) I feel like this announcement is too late. The biggest thing Heron offer is raw scale but since they decided to use existing Storm API, it has the same shitty spout /bolt API that Storms offer. In contrast, Spark streaming/Flink/ Kafka Streaming are all offering map/flatmap/filter/sink based functional API. At twitter most teams used SummingBird on top of Heron to get the same functional API but summingbird didnt get a lot of traction outside twitter and I am not sure how actively maintained OSS version of summingbird is. Even if you bite the bullet and decide to use SB with Heron, you will still miss out on a lot of usecases as SB was mostly focused on doing read/transform/aggregate/write whereas most streaming problem that i have noticed outside of twitter involve doing read/transform/aggregate/decision/write. I suppose you can implement decisioning in SB but i havent seen it done.

Comparing Heron to google millwheel is interesting because of the design choices they made. Heron support at least one and at most once message guarantees but at Twitter most job ran with acked turned off so it was at most once with acknowledged data loss ( they had a batchjob doing mop up work to pick up missing data). Google on the other hand implemented exactly once semantic by doing idempotent sinks/ watermarking and managing out of order messages plus deduping support. Since both Flink and Spark will be implementing Apache Beam (millwheel's predecessors) model, only reason I see someone picking heron instead of Flink/Spark is that they are operating at massive scale that flink/spark dont support yet

Re: Open-Sourcing Twitter Heron

#12

Post by eldenbishop » Wed, May 25, 2016, 5:57 PM UTC

scaleout1 wrote: post #11

As someone who has used Heron (along with MillWheel, Spark Streaming and Storm) I feel like this announcement is too late. The biggest thing Heron offer is raw scale but since they decided to use existing Storm API, it has the same shitty spout /bolt API that Storms offer. In contrast, Spark streaming/Flink/ Kafka Streaming are all offering map/flatmap/filter/sink based functional API. At twitter most teams used Summ…

Storm is a low level system for managing (optionally) transactional multi-machine tasks. It makes no assumptions about what is being processed (ie. analytics, data transforms). The primitives you are talking about exist in the child project Trident which runs on top of storm. Storm itself is no more for analytics than a web-server. It is a lower level tool.

Re: Open-Sourcing Twitter Heron

#13

Post by weego » Wed, May 25, 2016, 6:19 PM UTC

eldenbishop wrote: post #12

scaleout1 wrote: post #11

As someone who has used Heron (along with MillWheel, Spark Streaming and Storm) I feel like this announcement is too late. The biggest thing Heron offer is raw scale but since they decided to use existing Storm API, it has the same shitty spout /bolt API that Storms offer. In contrast, Spark streaming/Flink/ Kafka Streaming are all offering map/flatmap/filter/sink based functional API. At twitter most teams used Summ…

Storm is a low level system for managing (optionally) transactional multi-machine tasks. It makes no assumptions about what is being processed (ie. analytics, data transforms). The primitives you are talking about exist in the child project Trident which runs on top of storm. Storm itself is no more for analytics than a web-server. It is a lower level tool.

The parent also ignored the time-to-process difference which is drastically lower in storm. It has its flaws but scale is not the only metric to use as a decider

Re: Open-Sourcing Twitter Heron

#14

Post by jsmthrowaway » Wed, May 25, 2016, 6:24 PM UTC

mark242 wrote: post #3

How does Heron compare to Spark streaming?

Well, the big one is that Heron is built to support real-time streaming while Spark Streaming is not, given its choice to use micro-batching. If latency matters, that matters to you.

I'd appreciate if it were called Spark Microbatching, but we can't have everything.

Re: Open-Sourcing Twitter Heron

#15

Post by RBerenguel » Wed, May 25, 2016, 6:27 PM UTC

scaleout1 wrote: post #11

As someone who has used Heron (along with MillWheel, Spark Streaming and Storm) I feel like this announcement is too late. The biggest thing Heron offer is raw scale but since they decided to use existing Storm API, it has the same shitty spout /bolt API that Storms offer. In contrast, Spark streaming/Flink/ Kafka Streaming are all offering map/flatmap/filter/sink based functional API. At twitter most teams used Summ…

I had a look at the open source SummingBird as a possible way to implement a (soft) real time project I have, because I'm not specially Java-ish and Storm does not seem to play that nice with Scala (I've been told it works decently with Clojure, though, that might have been a solution to my non-Javaness) and it looked somewhat stale.

Ditched it and decided to do it in Spark with Scala (making it a good excuse to learn Scala). With so many real time options popping up and around, deciding which to pick is getting harder and harder.

Re: Open-Sourcing Twitter Heron

#16

Post by baby » Wed, May 25, 2016, 6:32 PM UTC

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

Re: Open-Sourcing Twitter Heron

#17

Post by drinkabeer » Wed, May 25, 2016, 6:36 PM UTC

baby wrote: post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

I think predictable performance is a major reason. Also, there are loads of developers which are still using Java, so you don't need to worry about expanding your teams and teaching hordes of developers a new development setup.

Re: Open-Sourcing Twitter Heron

#18

Post by zintinio5 » Wed, May 25, 2016, 6:38 PM UTC

baby wrote: post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

It has good tooling, IDE support, lots of people know it, it's fast, relatively easy to write, accessible from Scala, Groovy, Clojure, etc. Java is a step removed from having to write a system in C or C++, but you don't need absolute control over every single detail (garbage collection is acceptable). Most development happens in Java, C#, or C++, just not on HN.

There's something of an inertial effect with Java, where people continue to write Java because everyone else writes it (and thus it continually improves). What other platforms do you know of that have a similar track-record of performing in large-scale software?

Re: Open-Sourcing Twitter Heron

#19

Post by abritinthebay » Wed, May 25, 2016, 6:46 PM UTC

baby wrote: post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

It's not bad once you strip out most of the crap that it tries to foist upon you.

But honestly...? It's taught in schools a lot so more people are familiar with it. The greatest disservice to an entire generation of programmers is Java being standardized at Universities.

Re: Open-Sourcing Twitter Heron

#20

Post by EdSharkey » Wed, May 25, 2016, 6:46 PM UTC

baby wrote: post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

As I understand it, Twitter is heavily invested in Java and the JVM. They maintain their own fork of the JVM that they tweak for low latency. As a starting point for speed and stability, the JVM is top of the line.

I like coding with Java 8. There's lots of new syntactic sugar that let you cut down on all the boilerplate and cool new standard library features like Java 8 streams that let you do functional programming. The best part of Java 8 streams is they perform really well. You get more readable code along with better performance.