Live data from Hacker News

Using DTrace to measure Erlang dirty scheduler overhead

medium.com

11–20 of 20 posts

Re: Using DTrace to measure Erlang dirty scheduler overhead

#11

Here is an overview of how BEAM is implemented for people who want a quick refresher: http://www.erlang.org/euc/08/euc_smp.pdf And a comparison to the JVM here: http://ds.cs.ut.ee/courses/course-files/To303nis%20Pool%20.p...

Not sure why this JVM comparison is pulled all the time, from anyone with a little JVM experience it is clear the author has not a lot of knowledge of the JVM. Please don't.

Sent the author of the paper some mails, but sadly got no reply.

Re: Using DTrace to measure Erlang dirty scheduler overhead

#12
post #9
post #8

Earlier quoted context omitted.

The JVM could have had like 99% of all the VM market by now had Sun just opted to fix two things back in the day: * GC intrinsics, so you could implement functional languages easily. * Tail calls, so you could implement functional languages easily. I note LLVM made the same mistake :)

The JVM has nearly 99% of the non-Windows server VM market (you can't beat MS on Windows). And tail calls are coming once they matter enough to the users. What do you mean by GC intrinsics?

I'm talking about getting academia onboarded back in 1996 here. If you look at the industrial market, then sure, but I'm looking at what it takes to get languages that doesn't look like the normal imperative piece of crap to run on the JVM, and without tail-calls that is just hoop-jumping.

As for the GC, the interplay is the ability to tell the runtime where your pointers are. It is one of the places which usually lacks because many VMs assume a calling convention, like the one in Java, say. The JVM is a bit better here as it uses a stack-based engine, and as such is somewhat simpler to handle.

Functional compilers rarely, if ever, uses the standard conventions for handling this. Especially if they want to avoid boxing polymorphic parameters, and expand them for speed.

We could have had far better client-side and academic penetration of the JVM by now, had Sun played their cards differently. But alas, they didn't, and we are stuck with a Server VM market only.

Re: Using DTrace to measure Erlang dirty scheduler overhead

#13
post #12
post #9

Earlier quoted context omitted.

The JVM has nearly 99% of the non-Windows server VM market (you can't beat MS on Windows). And tail calls are coming once they matter enough to the users. What do you mean by GC intrinsics?

I'm talking about getting academia onboarded back in 1996 here. If you look at the industrial market, then sure, but I'm looking at what it takes to get languages that doesn't look like the normal imperative piece of crap to run on the JVM, and without tail-calls that is just hoop-jumping. As for the GC, the interplay is the ability to tell the runtime where your pointers are. It is one of the places which usually la…

While I'm sympathetic to bashing non-functional stuff, the JVM holds the record for languages. The CLR got the tail call thing partially right (AFAIK they didn't work well and weren't guaranteed) but due to the Windows only aspect MS chose, got almost no 3rd party language support.

The client side issue was much more to do with the terrible UI stuff and insistence on cross platform (hint: actual end users rarely care about cross platform since they tended to only run one platform at any time.) I believe MS was particularity enamored with Java but Sun slapped them away.

Client side was not lost due to lack of functional support. This should be trivially verifiable by the fact that fp is still only lightly used in industry.

Re: Using DTrace to measure Erlang dirty scheduler overhead

#14
post #12
post #9

Earlier quoted context omitted.

The JVM has nearly 99% of the non-Windows server VM market (you can't beat MS on Windows). And tail calls are coming once they matter enough to the users. What do you mean by GC intrinsics?

I'm talking about getting academia onboarded back in 1996 here. If you look at the industrial market, then sure, but I'm looking at what it takes to get languages that doesn't look like the normal imperative piece of crap to run on the JVM, and without tail-calls that is just hoop-jumping. As for the GC, the interplay is the ability to tell the runtime where your pointers are. It is one of the places which usually la…

When you say academia you mean academic PL research, which makes up a very small portion of CS academia. Most CS people care about FP just as much as the industry does, and the JVM is quite popular in the algorithmic fields (maybe not as much as C, but more than any other managed runtime).

Aside from tail-calls (which will come once people really ask for them), the JVM is about to have pretty much everything functional languages can ever need (value types and excellent box-elision), and the new JIT (Graal) is the biggest breakthrough in compiler technology in the last decade or so. PL academics are drooling over it. ECOOP had a full-day workshop dedicated just to Graal.

And you don't want to change calling conventions, because language interoperability is one of the JVM's greatest strengths. Here it is running JS, R and C, in the same REPL with Graal: https://dl.dropboxusercontent.com/u/292832/useR_multilang_de...

Re: Using DTrace to measure Erlang dirty scheduler overhead

#15
post #3

The 3-5 usec overhead, with VM optimization flag, for dirty schedulers is pretty good. What's the overhead of just passing the data through to a regular NIF? Probably gets burried in the jitter caused by cache and memory access times... (For others, if you don't know about the Erlang VM, and didn't understand the first couple of paragraphs, dirty schedulers is a new feature that solves the problem of running user cre…

Do Java Native Interfaces suffer the same sort of problem?

Re: Using DTrace to measure Erlang dirty scheduler overhead

#16

Here is an overview of how BEAM is implemented for people who want a quick refresher: http://www.erlang.org/euc/08/euc_smp.pdf And a comparison to the JVM here: http://ds.cs.ut.ee/courses/course-files/To303nis%20Pool%20.p...

Not sure why this JVM comparison is pulled all the time, from anyone with a little JVM experience it is clear the author has not a lot of knowledge of the JVM. Please don't. Sent the author of the paper some mails, but sadly got no reply.

Do you have a better article to point to? I'm eager to read about it!

Re: Using DTrace to measure Erlang dirty scheduler overhead

#17
post #15
post #3

The 3-5 usec overhead, with VM optimization flag, for dirty schedulers is pretty good. What's the overhead of just passing the data through to a regular NIF? Probably gets burried in the jitter caused by cache and memory access times... (For others, if you don't know about the Erlang VM, and didn't understand the first couple of paragraphs, dirty schedulers is a new feature that solves the problem of running user cre…

Do Java Native Interfaces suffer the same sort of problem?

Java maps its threads to native OS threads one-to-one. So if you execute in a thread in Java then jump to C you'll block that particular thread. Erlang implementas N:M concurrency where N Erlang processes map to M schedulers (a scheduler here is a separate OS thread ). N is usually greater than M. So you might have N=100K processes mapping to M=16 threads. If one of those processes makes a call to a C module it might block large (say 10k) number of other processes from executing.

Re: Using DTrace to measure Erlang dirty scheduler overhead

#18
post #7
post #3

The 3-5 usec overhead, with VM optimization flag, for dirty schedulers is pretty good. What's the overhead of just passing the data through to a regular NIF? Probably gets burried in the jitter caused by cache and memory access times... (For others, if you don't know about the Erlang VM, and didn't understand the first couple of paragraphs, dirty schedulers is a new feature that solves the problem of running user cre…

With jitter, the call time for a constant, in a dynamically linked library, is around 1200ns: enacl_nif:crypto_box_ZEROBYTES/0 value ------------- Distribution ------------- count 1000 | 0 1100 |@@@@ 110903 1200 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 879765 1300 | 7131 1400 | 689 1500 | 218 1600 | 45 1700 | 29

Thanks. That is a bit higher than I would expect but not too bad.

Re: Using DTrace to measure Erlang dirty scheduler overhead

#19
post #16

Earlier quoted context omitted.

Not sure why this JVM comparison is pulled all the time, from anyone with a little JVM experience it is clear the author has not a lot of knowledge of the JVM. Please don't. Sent the author of the paper some mails, but sadly got no reply.

Do you have a better article to point to? I'm eager to read about it!

No sorry, haven't found one. Most articles are either written by Erlang or by Java evangelists. I would wish for a comparision from someone with experience in both and without an agenda.

Re: Using DTrace to measure Erlang dirty scheduler overhead

#20
post #8
post #4

Earlier quoted context omitted.

I'll just note that the JVM part in that comparison is wrong. The main difference between BEAM and the JVM is that BEAM implements a much larger part of the language's functionality in the runtime, while -- at least for many JVM languages -- that is not the case with the JVM. The JVM -- similarly to the CPU+OS -- directly offers a rather general programming model -- shared memory, kernel threads etc., only with the a…

The JVM could have had like 99% of all the VM market by now had Sun just opted to fix two things back in the day: * GC intrinsics, so you could implement functional languages easily. * Tail calls, so you could implement functional languages easily. I note LLVM made the same mistake :)

> I note LLVM made the same mistake :)

The tail call problem is a problem of interoperability with C ABIs, not a "mistake" in LLVM. If LLVM hadn't made the decisions that it did, it wouldn't have gotten any traction at all, since you wouldn't be able to link LLVM code with any other libraries (including the system ones).

Post reply on HN