I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?
Open-Sourcing Twitter Heron
21–30 of 65 posts
Re: Open-Sourcing Twitter Heron
#22I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?
Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Scala nicer than both). These days, it comes down to whether the JVM memory overhead is a big deal on the specific project, and if not, for the class of projects discussed in the previous paragraph, I'm probably choosing Scala (but if my teammates object, then it's back to Java).
Re: Open-Sourcing Twitter Heron
#23They have dropped clojure completely. All the critical path messaging in Storm was done using clojure. Dropped netty too. The actual messaging ("stream manager") seems to be C++. Perhaps that explains the latency and CPU improvements. The architectural changes mentioned account for better cluster utilization, fault tolerance and back pressure implementation, but don't explain why raw streaming performance is so much…
Their default event loop implementation uses libevent, and they're using protobuf in some of their higher-level networking classes, but the networking code itself seems to pretty much be plain sockets (with a thin portability layer on top in a few places).
Re: Open-Sourcing Twitter Heron
#24Wonder how this compares to Storm 1.0
Re: Open-Sourcing Twitter Heron
#25I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?
Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…
IOW, "Let's say you want largely random pauses to impact latency non-deterministically." I'm not knocking GC, but it has serious consequences in systems that make this a weird assumption with which to start, particularly the "big" GC languages that you named. Scala in particular with its immutability generates a lot of garbage if you are not careful, and we were constantly fighting GC pressure (and, indirectly, our achievable concurrency and efficiency) at a household name Scala app. The number of JVM developers who are aware of this and capable with the memory subsystem -- i.e., off-heap strategies such as that used in Flink (relevantly) and some of the clever speedups in netty -- are dwarfed by the number of JVM developers in the whole, so you really need someone who understands these problems to efficiently scale a JVM language.
There are better systems languages if you can move away from GC, such as Rust. And modern C++ is fantastic for systems, and it's too bad it gets overlooked because of bam, your first assumption there. Most systems at Google are in C++ because they invested into the infrastructure that you need to support it. It should tell you something that they're heavily involved in each new version of C++.
(I'm talking about systems, not applications. GC is a viable tradeoff for programmer productivity in applications.)
Re: Open-Sourcing Twitter Heron
#26I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?
Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…
C# runs absolutely fine on nix.
Re: Open-Sourcing Twitter Heron
#27I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?
Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…
- Erlang (this fits your criteria at least as Java...)
- OCaml (concurrency isn't the best)
- Haskell (also fits your criteria very well)
- D (easy to use C libraries and sometimes C++)
- Rust (not GCed, but why is GCed a requirement?)
- Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else)
Java only seems like a good choice if you ignore all the languages that are a better choice than Java. IMO Erlang seems like the choice here (or Elixir if you don't like Erlang syntax).
Re: Open-Sourcing Twitter Heron
#28Earlier quoted context omitted.
Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…
> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…
After that, what's left?
At the moment, none of the above languages have any significant talent pool to draw from. If you're operating at "Twitter-scale" (I guess that's now a thing), you also need to be able to spin up a team of people that can get to work quickly without having to learn a language as they go. That eliminates a great number of languages just for logistical reasons. Rust might get there (and I hope it does!), but it's not there yet.
Maybe Twitter already has a significant number of skilled Java engineers, so they decided to use what they already knew really well. There's nothing wrong with that. Choosing any of the other languages you listed would have been a far riskier strategy.
Re: Open-Sourcing Twitter Heron
#29Earlier quoted context omitted.
Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…
> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…
- OP mentioned performance. Erlang is terrible for
CPU-bound tasks.- Haskell
- OP said non-esoteric.
- Rust - There's a significant ramp up before you learn to deal with the compiler and lifetimes.Re: Open-Sourcing Twitter Heron
#30Earlier quoted context omitted.
Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…
> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…
With Java, you can directly import a lot more, especially since the vast majority of OSS in this ecosystem is written in a JVM language.