Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-Sourcing Twitter Heron

blog.twitter.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Open-Sourcing Twitter Heron

#31

Post by hansjorg »

PeCaN wrote: post #27
fizx wrote: post #22

Earlier quoted context omitted.

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

I think you're underestimating the value of the enormous Java/JVM ecosystem.

That said, I'm hoping to deploy my first Rust project to production soon. It's come a long way for being such a young language.

Re: Open-Sourcing Twitter Heron

#32

Post by zintinio5 »

gshx wrote: post #21
baby wrote: post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

Go-lang is good but not as mature. Other than that, there's C++ and Erlang/Elixir. Not a whole lot of mature choices out there. For most of the critical production grade systems that need to be secure, have to scale and perform predictably, you don't want to blaze the trail.

I think your last point hits the nail on the head. There's a large amount of risk (especially at Twitter's size), and nobody wants to be saddled with an enormous and embarrassing failure. In addition, you need many hands to build the system, and finding developers for that ALSO have relevant experience could be difficult. You could make arguments that developers in are 10x as productive as your typical Java dev, but few people are going to take that chance. I'd say even Scala is still considered bleeding-edge at a lot of companies.

Re: Open-Sourcing Twitter Heron

#33

Post by fizx »

jsmthrowaway wrote: post #25
fizx wrote: post #22

Earlier quoted context omitted.

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> Let's say you want a GC'd language (no C, C++) IOW, "Let's say you want largely random pauses to impact latency non-deterministically." I'm not knocking GC, but it has serious consequences in systems that make this a weird assumption with which to start, particularly the "big" GC languages that you named. Scala in particular with its immutability generates a lot of garbage if you are not careful, and we were consta…

I'm excited for Rust for the reasons you talk about, but when I tried it a year ago, FFI was broken. I'd like to find a reason to retry it soon!

Re: Open-Sourcing Twitter Heron

#34

Post by kod »

jsmthrowaway wrote: post #14
mark242 wrote: post #3

How does Heron compare to Spark streaming?

Well, the big one is that Heron is built to support real-time streaming while Spark Streaming is not, given its choice to use micro-batching. If latency matters, that matters to you. I'd appreciate if it were called Spark Microbatching, but we can't have everything.

The amount of FUD thrown around about micro batching is really kind of silly.

How many streaming analytics use cases are ok with the JVM, ok with 10s of millis of latency, but not ok with 100s of millis of latency?

Re: Open-Sourcing Twitter Heron

#35

Post by mark242 »

jsmthrowaway wrote: post #14
mark242 wrote: post #3

How does Heron compare to Spark streaming?

Well, the big one is that Heron is built to support real-time streaming while Spark Streaming is not, given its choice to use micro-batching. If latency matters, that matters to you. I'd appreciate if it were called Spark Microbatching, but we can't have everything.

Okay, the better question then would be, how does this compare with writing a processor on top of Akka streams?

Re: Open-Sourcing Twitter Heron

#36

Post by jsmthrowaway »

kod wrote: post #34
jsmthrowaway wrote: post #14

Earlier quoted context omitted.

Well, the big one is that Heron is built to support real-time streaming while Spark Streaming is not, given its choice to use micro-batching. If latency matters, that matters to you. I'd appreciate if it were called Spark Microbatching, but we can't have everything.

The amount of FUD thrown around about micro batching is really kind of silly. How many streaming analytics use cases are ok with the JVM, ok with 10s of millis of latency, but not ok with 100s of millis of latency?

Remove "analytics" from your statement and you'll arrive at the answer, because while analytics is driving the streaming space as we speak it is literally one of the most boring applications of streaming imaginable. I positively cannot get excited about engagement numbers from an event pipeline. Intrusion detection, fraud detection, about half of infrastructure monitoring tasks, event sourcing... You're also glossing over the impact of microbatching which looks more like seconds than milliseconds, as well as what microbatching does to your windowing abilities (such as having to double-process to work around the microbatch interval as applied to your desired windowing semantics).

Microbatch latency immediately rules out several useful applications of streaming. I also didn't say anything about the JVM nor it being okay for my purposes. You did.

I can back up my statements on streaming from Flink to Samza to Dataflow/Beam to Storm to MillWheel to Spark "Streaming" and back again because it has been my primary focus (literally thinking of nothing else) for a couple years. Please accuse someone else of FUD because that's a relatively veiled way to say "you don't know what you're talking about," and I assure you that you're (condescendingly) wrong. I think you're also coming at that angle from interpreting me as negative on Spark Streaming. Read carefully.

Re: Open-Sourcing Twitter Heron

#37

Post by PeCaN »

flippant wrote: post #29
PeCaN wrote: post #27

Earlier quoted context omitted.

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

- Erlang/Elixir - OP mentioned performance. Erlang is terrible for CPU-bound tasks. - Haskell - OP said non-esoteric. - Rust - There's a significant ramp up before you learn to deal with the compiler and lifetimes.

> Erlang

Erlang's CPU performance is not bad IME (though admittedly I can't think of anything I've implemented in both Erlang and Java to compare). But that's beside the point; you pick Erlang for networked and I/O bound tasks. Twitter mentioned running Heron on several hundred machines—at that scale, single-thread performance starts to matter less and Erlang's scaling efficiency closes the gap.

> Haskell

Haskell may have been esoteric in 2005, but it's hardly so anymore. (Also, I mention Mercury, and Haskell is the language you decide is too esoteric‽)

> Rust

So you're saying they wouldn't use it because they didn't know it. I suspect that's the real reason they chose Java—not because it is necessarily the best language suited for the task, but because it's the best of the languages Heron's developers knew.

Re: Open-Sourcing Twitter Heron

#38

Post by zintinio5 »

JonathonW wrote: post #23
lovelearning wrote: post #9

They have dropped clojure completely. All the critical path messaging in Storm was done using clojure. Dropped netty too. The actual messaging ("stream manager") seems to be C++. Perhaps that explains the latency and CPU improvements. The architectural changes mentioned account for better cluster utilization, fault tolerance and back pressure implementation, but don't explain why raw streaming performance is so much…

Looks like a homegrown networking framework: https://github.com/twitter/heron/tree/master/heron/common/sr... Their default event loop implementation uses libevent, and they're using protobuf in some of their higher-level networking classes, but the networking code itself seems to pretty much be plain sockets (with a thin portability layer on top in a few places).

It's some...interesting code.

https://github.com/twitter/heron/blob/master/heron/common/sr...

Re: Open-Sourcing Twitter Heron

#39

Post by PeCaN »

hansjorg wrote: post #31
PeCaN wrote: post #27

Earlier quoted context omitted.

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

I think you're underestimating the value of the enormous Java/JVM ecosystem. That said, I'm hoping to deploy my first Rust project to production soon. It's come a long way for being such a young language.

> I think you're underestimating the value of the enormous Java/JVM ecosystem.

You could be right. For a number of the languages I mentioned I banked on really good FFIs allowing easy usage of C libraries. Perhaps in some domains, including stream processing, there are more and better Java or C++ libraries for them to build on.

Re: Open-Sourcing Twitter Heron

#40

Post by untog »

zintinio5 wrote: post #38
JonathonW wrote: post #23

Earlier quoted context omitted.

Looks like a homegrown networking framework: https://github.com/twitter/heron/tree/master/heron/common/sr... Their default event loop implementation uses libevent, and they're using protobuf in some of their higher-level networking classes, but the networking code itself seems to pretty much be plain sockets (with a thin portability layer on top in a few places).

It's some...interesting code. https://github.com/twitter/heron/blob/master/heron/common/sr...

So that's what Twitter has been working all this time
Post reply on HN