Open-Sourcing Twitter Heron • Hacker News Board

Re: Open-Sourcing Twitter Heron

#61

Post by kod » Thu, May 26, 2016, 4:32 PM UTC

jsmthrowaway wrote: post #36

kod wrote: post #34

Earlier quoted context omitted.

The amount of FUD thrown around about micro batching is really kind of silly. How many streaming analytics use cases are ok with the JVM, ok with 10s of millis of latency, but not ok with 100s of millis of latency?

Remove "analytics" from your statement and you'll arrive at the answer, because while analytics is driving the streaming space as we speak it is literally one of the most boring applications of streaming imaginable. I positively cannot get excited about engagement numbers from an event pipeline. Intrusion detection, fraud detection, about half of infrastructure monitoring tasks, event sourcing... You're also glossing…

> You're also glossing over the impact of microbatching which looks more like seconds than milliseconds

I've run spark streaming jobs at 250ms batch times. This comment you made, right here, is why I used the word FUD.

Re: Open-Sourcing Twitter Heron

#62

Post by abc_lisper » Thu, May 26, 2016, 5:48 PM UTC

jsmthrowaway wrote: post #44

scaleout1 wrote: post #11

As someone who has used Heron (along with MillWheel, Spark Streaming and Storm) I feel like this announcement is too late. The biggest thing Heron offer is raw scale but since they decided to use existing Storm API, it has the same shitty spout /bolt API that Storms offer. In contrast, Spark streaming/Flink/ Kafka Streaming are all offering map/flatmap/filter/sink based functional API. At twitter most teams used Summ…

> they are operating at massive scale that flink/spark dont support yet Flink certainly scales just fine, for what it's worth. Flink 1.0 is quite good, and I'd consider what I'm doing "massive scale"; the ease of 1MM+ QPS with decent p95 latency via Flink surprised me compared to other systems that I investigated in this space. Most hip-fired benchmarks, including that awful Yahoo! one that everybody cites, use Flink…

> Most hip-fired benchmarks, including that awful Yahoo! one that everybody cites, use Flink poorly

Why are Yahoo! benchmarks awful? How did they manage to use Flink poorly?

For people who don't know what he is referring to, check this: https://yahooeng.tumblr.com/post/135321837876/benchmarking-s...

Re: Open-Sourcing Twitter Heron

#63

Post by jedisfullofit » Fri, May 27, 2016, 12:02 AM UTC

jsmthrowaway wrote: post #25

fizx wrote: post #22

Earlier quoted context omitted.

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> Let's say you want a GC'd language (no C, C++) IOW, "Let's say you want largely random pauses to impact latency non-deterministically." I'm not knocking GC, but it has serious consequences in systems that make this a weird assumption with which to start, particularly the "big" GC languages that you named. Scala in particular with its immutability generates a lot of garbage if you are not careful, and we were consta…

"Scala in particular with its immutability generates a lot of garbage if you are not careful, and we were constantly fighting GC pressure (and, indirectly, our achievable concurrency and efficiency) at a household name Scala app. The number of JVM developers who are aware of this and capable with the memory subsystem -- i.e., off-heap strategies such as that used in Flink (relevantly) and some of the clever speedups in netty -- are dwarfed by the number of JVM developers in the whole, so you really need someone who understands these problems to efficiently scale a JVM language."

Jed, once again you are laying claim to experience you do not have. If you ever once had touched Scala at Foursquare it would be a different story. This is just a bunch of tired platitudes.

Re: Open-Sourcing Twitter Heron

#64

Post by technion » Fri, May 27, 2016, 4:13 AM UTC

pron wrote: post #50

PeCaN wrote: post #27

Earlier quoted context omitted.

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

Aside from Erlang which is far too slow (most Erlang apps use C for the data plane) the others are far too risky choices in terms of hiring, future support and general maintainability. Also there are far too many unknowns about them. Rust shows a lot of promise and may very well be suitable for this kind of things, but it may still be too young for an important infrastructure project at Twitter. It is also harder to…

    Aside from Erlang which is far too slow (most Erlang apps use C for the data plane)

I'm going to call "citation needed" on that. Riak and EJabberd have a reputation for scale and performance, and the only C I can see referenced is where existing libraries like Zlib get used.

Edit: You are still right. The hiring issues are relevant, and apply to Erlang also.

Re: Open-Sourcing Twitter Heron

#65

Post by lpw25 » Fri, May 27, 2016, 6:36 AM UTC

PeCaN wrote: post #27

fizx wrote: post #22

Earlier quoted context omitted.

Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left? Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Sca…

> what's left? - Erlang (this fits your criteria at least as Java...) - OCaml (concurrency isn't the best) - Haskell (also fits your criteria very well) - D (easy to use C libraries and sometimes C++) - Rust (not GCed, but why is GCed a requirement?) - Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else) Java only seems like a good choice if you ignore all the language…

> OCaml (concurrency isn't the best)

Pet peeve: OCaml's concurrency is pretty good, it's parallelism that it struggles with.