Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

ds.cs.ut.ee

Post reply on HN
Print view View original

21–30 of 60 posts

Re: Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

#21

Post by jlouis »

pron wrote: post #4

The report mentions Quasar (I'm its main author) and simply says, "instrumentation has many challenges that add complexity to the program instead of removing it." -- without mentioning why. I believe this is false. Quasar removes just as much complexity as Erlang does (Erlang's enforcement of immutability is orthogonal; indeed, if you use Quasar with Clojure you get that, too), but it is true that the instrumentation…

Maybe I should make the argument as to why ETS is not garbage collected. It is a pragmatic solution to an often occurring problem in software.

You have a lot of data.

You rarely change that data.

You still have to walk over it when you garbage collect.

ETS allows you to store Erlang terms into a tuple space outside the heap of any process. This means you avoid garbage collecting, and you can have 120 gigabyte of RSS, but only have to collect 400 megabytes of those. If you look at how many "mmap" implementations there are for GC'ed languages, and if you have ever reached for one, you know what I'm talking about.

Re: Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

#22

Post by jlouis »

pjmlp wrote: post #18
jlouis wrote: post #17

Earlier quoted context omitted.

Take the fastest webserver written for Java and for Erlang. Now run static requests around 2048 bytes, 10k connections, 3 req/s on each connection for 5 minutes. Measure the 99.9th percentile latency. Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. This is not desirable in a soft-realtime system. I'd definitely not order one as the strict superset of the other.

Try that with Zing. The beauty of the JVM, is that by being a specification, there are lots of implementations to choose from. Yet people seem to think OpenJDK is the only one.

I'm aware of Zing and I'm pretty sure it would fare way better in this kind of problem. However, I'm inclined to guess that every time you sink 1 hour of work into BEAM, there is somewhere between 5-15 hours sunk into OpenJDK and other VMs in Java space. It wins many competitions by brute force alone.

Furthermore, Erlang has some things against it in the speed department:

It is forced functional.

It is dynamically typed (A tracing JIT can somewhat alleviate this problem).

But I think the BEAM architecture is pretty sound. It has a lot of things in common with a micro kernel, architecturally. And while micro kernels are not popular currently, they see a lot of use in the embedded space, on sattelite's and so on.

Re: Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

#23

Post by pron »

phamilton wrote: post #11
pron wrote: post #4

The report mentions Quasar (I'm its main author) and simply says, "instrumentation has many challenges that add complexity to the program instead of removing it." -- without mentioning why. I believe this is false. Quasar removes just as much complexity as Erlang does (Erlang's enforcement of immutability is orthogonal; indeed, if you use Quasar with Clojure you get that, too), but it is true that the instrumentation…

Regarding the JVM as a strict superset, there are a few features I'm not sure fit that model. Isolated garbage collection. Limiting GC to a single process/thread. It's my understanding that this is only possible with individual heaps, prohibiting shared memory. I see this as a polarizing tradeoff (with pros and cons). I don't think the JVM does this, and therefore isn't a superset. Preemptive scheduling and soft real…

> It's my understanding that this is only possible with individual heaps, prohibiting shared memory.

Yes, but the only advantage is a simpler GC algorithm. HotSpot's GCs are so advanced (let alone HotSpot descendents like Zing) that they give you similar behavior even without isolation, and they include collection of shared data structures, too.

> Would Quasar, for instance, pause a tight for loop if execution was running long? Or does it only pause on blocking operations like IO?

Quasar does full preemptive scheduling but not time-sliced based preemption. This is not because it can't -- as a matter of fact, early versions of Quasar did time-sliced based preemption, but it turned out to provide no benefit whatsoever. In fact, Erlang's behavior is a limitation. The reason is that Erlang has only one type of thread -- the process, or the user-mode thread -- and therefore has to handle any type of thread, including those that are computation heavy. Thing is, work-stealing is a great scheduling mechanism for transaction-serving threads (that block very often), but not so good for computational threads. Quasar lets you choose: fibers for transactions stuff, and plain threads for long-running computations, both are abstracted into what we call a "strand".

Re: Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

#24

Post by pron »

jlouis wrote: post #17
pron wrote: post #4

The report mentions Quasar (I'm its main author) and simply says, "instrumentation has many challenges that add complexity to the program instead of removing it." -- without mentioning why. I believe this is false. Quasar removes just as much complexity as Erlang does (Erlang's enforcement of immutability is orthogonal; indeed, if you use Quasar with Clojure you get that, too), but it is true that the instrumentation…

Take the fastest webserver written for Java and for Erlang. Now run static requests around 2048 bytes, 10k connections, 3 req/s on each connection for 5 minutes. Measure the 99.9th percentile latency. Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. This is not desirable in a soft-realtime system. I'd definitely not order one as the strict superset of the other.

> Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so.

Not even close (certainly not when using Quasar). Whether or not you have large GC pauses depends on how you use the heap. If you only allocate objects that live for the duration of the request (which can be enforced by your choice of JVM language) you get the same GC behavior as Erlang (only more general), and a much, much, much, better computation performance, due to HotSpot having one of the world's most advanced optimizing compilers.

Re: Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

#25

Post by [deleted] »

jlouis wrote: post #17
pron wrote: post #4

The report mentions Quasar (I'm its main author) and simply says, "instrumentation has many challenges that add complexity to the program instead of removing it." -- without mentioning why. I believe this is false. Quasar removes just as much complexity as Erlang does (Erlang's enforcement of immutability is orthogonal; indeed, if you use Quasar with Clojure you get that, too), but it is true that the instrumentation…

Take the fastest webserver written for Java and for Erlang. Now run static requests around 2048 bytes, 10k connections, 3 req/s on each connection for 5 minutes. Measure the 99.9th percentile latency. Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. This is not desirable in a soft-realtime system. I'd definitely not order one as the strict superset of the other.

[deleted]

Re: Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

#26

Post by jlouis »

[deleted] wrote: post #25
jlouis wrote: post #17

Earlier quoted context omitted.

Take the fastest webserver written for Java and for Erlang. Now run static requests around 2048 bytes, 10k connections, 3 req/s on each connection for 5 minutes. Measure the 99.9th percentile latency. Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. This is not desirable in a soft-realtime system. I'd definitely not order one as the strict superset of the other.

[deleted]

The TechEmpower Benchmarks measures the maximal number of requests per second you can churn through a server. They don't care for the number of connections, but simply pick the best result from trying different connection counts.

Also, they don't really measure latency correct I think, because they make the mistake of "coordinated omission" (Look it up, Gil Tene, of Zing/Azul fame is its main discoverer).

I was very careful in my wording. You need 10.000 connections, each running 3 req/s for 30.000 req/s. So we are not trying to maximize the system. We are trying to saturate it over a long period with a constant load. And you need to use wrk2 (also from Gil Tene) because otherwise the load generator and the System-under-test ends up coordinating among themselves, hiding the fact that some of the 10.000 connections are starved to get more req/s through on a subset.

The Java frameworks are very very good at handling 512 connections and throwing as many requests through those as possible. What they fail at, is 10.000 connections. What usually happens in bad load generators is that they allow the Java code to starve, say 9500 of those connections while getting all the good numbers on the last 500. wrk2 doesn't allow that to happen, and starvation is counted against the SUT.

As for the source, you would have to wait. I have the raw data, but I have not yet written the blog post. I think I've given you all the information you need to carry out the test if you are so inclined here.

Re: Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

#27

Post by abalone »

pron wrote: post #24
jlouis wrote: post #17

Earlier quoted context omitted.

Take the fastest webserver written for Java and for Erlang. Now run static requests around 2048 bytes, 10k connections, 3 req/s on each connection for 5 minutes. Measure the 99.9th percentile latency. Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. This is not desirable in a soft-realtime system. I'd definitely not order one as the strict superset of the other.

> Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. Not even close (certainly not when using Quasar). Whether or not you have large GC pauses depends on how you use the heap. If you only allocate objects that live for the duration of the request (which can be enforced by your choice of JVM language) you get the same GC behavior as Erlang (only more general), and a…

The TechEmpower Web Framework Benchmarks score Java 5X faster than Erlang at a simple JSON test. (There's an even simpler plaintext test but no one was implemented it for Erlang.)

However it's worth nothing the test only runs for 15 seconds. It might be an interesting addition to run it for 10 minutes and measure 99.9% latency, as jlouis proposes, and prove one of you correct.

https://www.techempower.com/benchmarks/#section=data-r10&hw=...

Re: Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

#28

Post by jlouis »

pron wrote: post #24
jlouis wrote: post #17

Earlier quoted context omitted.

Take the fastest webserver written for Java and for Erlang. Now run static requests around 2048 bytes, 10k connections, 3 req/s on each connection for 5 minutes. Measure the 99.9th percentile latency. Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. This is not desirable in a soft-realtime system. I'd definitely not order one as the strict superset of the other.

> Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. Not even close (certainly not when using Quasar). Whether or not you have large GC pauses depends on how you use the heap. If you only allocate objects that live for the duration of the request (which can be enforced by your choice of JVM language) you get the same GC behavior as Erlang (only more general), and a…

HotSpot doesn't matter that much in this test. You end up spending 80% of the time in the kernel and when I tested, the Erlang code only spent a fraction of it's time in the emulator loop. In other words, the speed comes from everything but the optimizing compiler. It is from the runtimes.

Re: Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

#29

Post by abalone »

jlouis wrote: post #26
[deleted] wrote: post #25

Earlier quoted context omitted.

[deleted]

The TechEmpower Benchmarks measures the maximal number of requests per second you can churn through a server. They don't care for the number of connections, but simply pick the best result from trying different connection counts. Also, they don't really measure latency correct I think, because they make the mistake of "coordinated omission" (Look it up, Gil Tene, of Zing/Azul fame is its main discoverer). I was very…

I deleted my comment because I realize the benchmarks don't cover a 5 minute test. It's unproven whether the Java implementations need to in fact GC over that time. I'm skeptical though -- they probably reuse all their objects, and I think it would be interesting to prove that.

Also, the benchmarks do in fact test all the way up to 16,384 simultaneous connections.[1]

I bet if you published a test case that proves Java GC impacting the max latency over 5 minutes, that would get the attention of TechEmpower and make a case for extending the test past 15 seconds.

[1] "The high-concurrency plaintext test type is tested at 256, 1,024, 4,096, and 16,384 client-side concurrency." http://frameworkbenchmarks.readthedocs.org/en/latest/Project...

Re: Comparison of Erlang Runtime System and Java Virtual Machine [pdf]

#30

Post by pjmlp »

dxhdr wrote: post #20
pjmlp wrote: post #18

Earlier quoted context omitted.

Try that with Zing. The beauty of the JVM, is that by being a specification, there are lots of implementations to choose from. Yet people seem to think OpenJDK is the only one.

Sure I'll just fork over $8,000 and get started... The beauty of BEAM is that I don't have to know or worry about projects like Zing, BEAM "just works." (Another mark against Zing just because you brought it up -- where's the source code?)

Not all business value FOSS.
Post reply on HN