Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-Sourcing Twitter Heron

blog.twitter.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Open-Sourcing Twitter Heron

#51

Post by notJim »

baby wrote: post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

I actually like Java pretty well. The language itself is mostly pretty OK, the tooling is super mature, it's typed which is huge. The community has a lot of smart people. Definitely not something super-exciting or my favorite language, but overall it's just pretty smooth to work with.

I'm curious what you found unpleasant about it.

Re: Open-Sourcing Twitter Heron

#52

Post by heavenlyhash »

scaleout1 wrote: post #11

As someone who has used Heron (along with MillWheel, Spark Streaming and Storm) I feel like this announcement is too late. The biggest thing Heron offer is raw scale but since they decided to use existing Storm API, it has the same shitty spout /bolt API that Storms offer. In contrast, Spark streaming/Flink/ Kafka Streaming are all offering map/flatmap/filter/sink based functional API. At twitter most teams used Summ…

Thank you for this comment. It contained orders of magnitude more useful information about the API choices and data models that define this system than the linked article itself.

Re: Open-Sourcing Twitter Heron

#53

Post by draw_down »

baby wrote: post #16

I'm genuinely curious as to why new products are constantly written in Java. My experience with the language is far from pleasant. Is it because people actually like the language? Is it because there is no other alternative when it comes to solid development? ...?

Well, sometimes other things matter than developer ergonomics.

Re: Open-Sourcing Twitter Heron

#54

Post by int_handler »

insulanian wrote: post #26
fizx wrote: post #22

Earlier quoted context omitted.

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> on nix (no C#) C# runs absolutely fine on nix.

Yes, it is possible to run C# on *nix using Mono but the ecosystem isn't as rich as that of Java.

Re: Open-Sourcing Twitter Heron

#55

Post by hyperpape »

pron wrote: post #50
PeCaN wrote: post #27

Earlier quoted context omitted.

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

Aside from Erlang which is far too slow (most Erlang apps use C for the data plane) the others are far too risky choices in terms of hiring, future support and general maintainability. Also there are far too many unknowns about them. Rust shows a lot of promise and may very well be suitable for this kind of things, but it may still be too young for an important infrastructure project at Twitter. It is also harder to…

Is there any kind of overview of the JIT improvements coming in Java 9, or do you have to follow the relevant mailing lists/commits?

Re: Open-Sourcing Twitter Heron

#56

Post by coredog64 »

jsmthrowaway wrote: post #42
snaky wrote: post #41

Earlier quoted context omitted.

> you really need someone who understands these problems to efficiently scale a JVM language Yes. That's not a problem to find and hire one or two of these guys to write a core. And then hire a ton of regular Java developers to write a ton of code you need around that core, educated and managed by those two smart guys. Compare it to the problem you have going C, C++, Rust, Nim way. Now you need to hire a ton of guys…

Your entire second paragraph indicates to me that you haven't spent time at a well-oiled C++ shop or potentially haven't worked with modern C++ at all. What does "crash every 5 seconds" even mean? Do you think outside of Java we are all sitting around pounding rocks together and dealing with segfaults all day long? Are you saying JVM languages don't NPE in poor hands? There are three focus areas of wrongness in your…

I think he is casting aspersions on the average programmer. I'm on a significant chunk of PRs across projects at my employer and I have seen crazy crazy shit from Java devs with years of experience. In some cases these same devs hold Java certifications from Sun/Oracle. I shudder to think of what would happen if they decided to have these guys write C or C++.

Re: Open-Sourcing Twitter Heron

#57

Post by pron »

hyperpape wrote: post #55
pron wrote: post #50

Earlier quoted context omitted.

Aside from Erlang which is far too slow (most Erlang apps use C for the data plane) the others are far too risky choices in terms of hiring, future support and general maintainability. Also there are far too many unknowns about them. Rust shows a lot of promise and may very well be suitable for this kind of things, but it may still be too young for an important infrastructure project at Twitter. It is also harder to…

Is there any kind of overview of the JIT improvements coming in Java 9, or do you have to follow the relevant mailing lists/commits?

I'm referring to Graal, which would be available in Java 9 as an alternative JIT to HotSpot's C2 compiler: https://wiki.openjdk.java.net/display/Graal/Publications+and...

Re: Open-Sourcing Twitter Heron

#58

Post by abiox »

PeCaN wrote: post #27
fizx wrote: post #22

Earlier quoted context omitted.

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

java isn't that bad. there's a lot of crappy code-culture that comes from the uber-enterprisey arena, but in more reasonable hands it's quite capable.

it's also worth nothing that Java comes with the JVM and Hotspot JIT, which earn it a bit of swagger.

Re: Open-Sourcing Twitter Heron

#59

Post by galistoca »

PeCaN wrote: post #37
flippant wrote: post #29

Earlier quoted context omitted.

- Erlang/Elixir - OP mentioned performance. Erlang is terrible for CPU-bound tasks. - Haskell - OP said non-esoteric. - Rust - There's a significant ramp up before you learn to deal with the compiler and lifetimes.

> Erlang Erlang's CPU performance is not bad IME (though admittedly I can't think of anything I've implemented in both Erlang and Java to compare). But that's beside the point; you pick Erlang for networked and I/O bound tasks. Twitter mentioned running Heron on several hundred machines—at that scale, single-thread performance starts to matter less and Erlang's scaling efficiency closes the gap. > Haskell Haskell may…

What is the definition of "best" anyway? If you really just think about pure performance nothing can beat a binary code probably. When you say "best" it probably means most productive in the short/long term. In that case it's not about what the language is good at. After all it's humans who use the language. Just like how esperanto is such an awesome concept but nobody uses it and that makes it a useless language, the more people use a language the more valuable it becomes. And yeah Haskell is esoteric. I am confident that I know way many more programming languages than an average programmer but I don't program in Haskell. Try picking 10 random programmers and ask them if they can program in Haskell, at best you'll find 1 (actually that's even being too optimistic)

Re: Open-Sourcing Twitter Heron

#60

Post by nstart »

hansjorg wrote: post #31
PeCaN wrote: post #27

Earlier quoted context omitted.

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

I think you're underestimating the value of the enormous Java/JVM ecosystem. That said, I'm hoping to deploy my first Rust project to production soon. It's come a long way for being such a young language.

Hey.. I'm just super curious to learn a lot about the Rust ecosystem. Would you be up for sharing any details of what your rust project might be? Thanks!
Post reply on HN