Live data from Hacker News

Open-Sourcing Twitter Heron

blog.twitter.com

21–30 of 65 posts

Re: Open-Sourcing Twitter Heron

#21
post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

Go-lang is good but not as mature. Other than that, there's C++ and Erlang/Elixir. Not a whole lot of mature choices out there. For most of the critical production grade systems that need to be secure, have to scale and perform predictably, you don't want to blaze the trail.

Re: Open-Sourcing Twitter Heron

#22
post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left?

Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Scala nicer than both). These days, it comes down to whether the JVM memory overhead is a big deal on the specific project, and if not, for the class of projects discussed in the previous paragraph, I'm probably choosing Scala (but if my teammates object, then it's back to Java).

Re: Open-Sourcing Twitter Heron

#23

They have dropped clojure completely. All the critical path messaging in Storm was done using clojure. Dropped netty too. The actual messaging ("stream manager") seems to be C++. Perhaps that explains the latency and CPU improvements. The architectural changes mentioned account for better cluster utilization, fault tolerance and back pressure implementation, but don't explain why raw streaming performance is so much…

Looks like a homegrown networking framework: https://github.com/twitter/heron/tree/master/heron/common/sr...

Their default event loop implementation uses libevent, and they're using protobuf in some of their higher-level networking classes, but the networking code itself seems to pretty much be plain sockets (with a thin portability layer on top in a few places).

Re: Open-Sourcing Twitter Heron

#24

Wonder how this compares to Storm 1.0

Why am I being down-modded? When Twitter first announced Heron(~ a year ago), they compared it with a version of Storm available then. Since then Storm has improved it's latency, ability to scale-out, added back-pressure and in some cases it is 10x faster than the previous version of storm (0.10.0?). I was wondering how Heron compares to Storm's performance now.

Re: Open-Sourcing Twitter Heron

#25
post #22
post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> Let's say you want a GC'd language (no C, C++)

IOW, "Let's say you want largely random pauses to impact latency non-deterministically." I'm not knocking GC, but it has serious consequences in systems that make this a weird assumption with which to start, particularly the "big" GC languages that you named. Scala in particular with its immutability generates a lot of garbage if you are not careful, and we were constantly fighting GC pressure (and, indirectly, our achievable concurrency and efficiency) at a household name Scala app. The number of JVM developers who are aware of this and capable with the memory subsystem -- i.e., off-heap strategies such as that used in Flink (relevantly) and some of the clever speedups in netty -- are dwarfed by the number of JVM developers in the whole, so you really need someone who understands these problems to efficiently scale a JVM language.

There are better systems languages if you can move away from GC, such as Rust. And modern C++ is fantastic for systems, and it's too bad it gets overlooked because of bam, your first assumption there. Most systems at Google are in C++ because they invested into the infrastructure that you need to support it. It should tell you something that they're heavily involved in each new version of C++.

(I'm talking about systems, not applications. GC is a viable tradeoff for programmer productivity in applications.)

Re: Open-Sourcing Twitter Heron

#26
post #22
post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> on nix (no C#)

C# runs absolutely fine on nix.

Re: Open-Sourcing Twitter Heron

#27
post #22
post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> what's left?

- Erlang (this fits your criteria at least as Java...)

- OCaml (concurrency isn't the best)

- Haskell (also fits your criteria very well)

- D (easy to use C libraries and sometimes C++)

- Rust (not GCed, but why is GCed a requirement?)

- Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else)

Java only seems like a good choice if you ignore all the languages that are a better choice than Java. IMO Erlang seems like the choice here (or Elixir if you don't like Erlang syntax).

Re: Open-Sourcing Twitter Heron

#28
post #27
post #22

Earlier quoted context omitted.

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

Let's also say that you want to be able to hire programmers to write and maintain it. And it would be nice to make it open to many more people so they could find and fix bugs.

After that, what's left?

At the moment, none of the above languages have any significant talent pool to draw from. If you're operating at "Twitter-scale" (I guess that's now a thing), you also need to be able to spin up a team of people that can get to work quickly without having to learn a language as they go. That eliminates a great number of languages just for logistical reasons. Rust might get there (and I hope it does!), but it's not there yet.

Maybe Twitter already has a significant number of skilled Java engineers, so they decided to use what they already knew really well. There's nothing wrong with that. Choosing any of the other languages you listed would have been a far riskier strategy.

Re: Open-Sourcing Twitter Heron

#29
post #27
post #22

Earlier quoted context omitted.

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

- Erlang/Elixir

  - OP mentioned performance. Erlang is terrible for 
CPU-bound tasks.

- Haskell

  - OP said non-esoteric.
- Rust

  - There's a significant ramp up before you learn to deal with the compiler and lifetimes.

Re: Open-Sourcing Twitter Heron

#30
post #27
post #22

Earlier quoted context omitted.

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

None of these languages even come close to the level of library support and popularity as Java. Additionally it becomes a people problem when you use a language people aren't as familiar with leading to people building connectors which don't perform as well as using the original libraries.

With Java, you can directly import a lot more, especially since the vast majority of OSS in this ecosystem is written in a JVM language.

Post reply on HN