A Deep Introduction to JIT Compilers: JITs are not very Just-in-time • Hacker News Board

Re: A Deep Introduction to JIT Compilers: JITs are not very Just-in-time

#61

Post by munificent » Mon, Jul 06, 2020, 1:51 AM UTC

kipply wrote: post #27

dialamac wrote: post #5

Earlier quoted context omitted.

> Thank to runtime information it can occasionally exceed performance of statically compiled language. Interestingly in 30 years I have not once heard of a case where this theoretical benefit has manifested as a clear advantage in any real world application when looking at the system as a whole... amdahls law and all that. You can always hand tune the 1-10% hotspots for reasonable cost most of the time, and even stat…

JITs vs static compilers on a language made for static compilers is probably a contest that JITs will never win. The keys to making JITs "better" than static compilation is (by my limited knowledge) creation of languages that are made to be JIT compiled (like Java) and ensuring that tuning JITs is easier than PGO (though nothing that is currently in development seems promising to this end)

This is exactly right.

You do see JITs winning in Java because Java's "everything is virtual" and "most for loop are interface calls to Iterator" makes it very hard to statically compile efficiently.

Languages designed for static compilation generally consider virtual dispatch to have a user-visible cost and don't unilaterally make users paying it without them asking for it.

Re: A Deep Introduction to JIT Compilers: JITs are not very Just-in-time

#62

Post by scarmig » Mon, Jul 06, 2020, 2:54 AM UTC

kipply wrote: post #10

cs702 wrote: post #9

Very nice. I just looked at the author's resume.[a] It appears she is still in, or only very recently graduated from, high school . Is that right? Impressive! [a] https://carolchen.me/

Yep, class of 2019, jit to actually have a physical prom and graduation

I see what you did there.

Great blog, and you're into aerials! Always exciting to see someone else here who gets their invert on.

Re: A Deep Introduction to JIT Compilers: JITs are not very Just-in-time

#63

Post by monadic2 » Mon, Jul 06, 2020, 3:17 AM UTC

saagarjha wrote: post #56

jkubrynski wrote: post #55

There is also an interesting presentation about how Azul implemented their JIT called Falcon, that is fully based on LLVM. https://www.youtube.com/watch?v=Uqch1rjPls8 Generally LLVM is a nice idea that allows vendors to reuse major components.

The one problem with LLVM is that it’s not very well suited to JIT compilers, a major component of which is its slowness.

That’s not really accurate. It has strict requirements about stack traversal required to appropriately trace memory roots. This is incidental to JIT vs AOT.

Re: A Deep Introduction to JIT Compilers: JITs are not very Just-in-time

#64

Post by tsimionescu » Mon, Jul 06, 2020, 3:57 AM UTC

darksaints wrote: post #42

saagarjha wrote: post #34

Earlier quoted context omitted.

Most JIT compilers will go after any code that shows up as hot, regardless of when it executes. If your API code is that, even a minute after startup, it should really be getting optimized…

Think of an use case where you have an API whose mode behavior is to do nothing, waiting for a call... but commonly gets calls that are very compute intense for short bursts of time. With the most common type of JITs, which profile once and compile once, I'm going to get code that is optimized for startup and initialization. If I have an "advanced" JIT, which is constantly deoptimizing and reoptimizing for whatever i…

The most commonly used JIT in the world is almost certainly the one in the OpenJDK, which is of the advanced kind. And it does not suffer from the problem you are talking about, because it only ever looks at code that is getting executed.

Basically, it will do something like: interpret a function for a the first few thousand times it is executed, collecting execution metrics. After a threshold is reached, next time that function is called, start compiling it, optimizing based on the execution metrics collected earlier. Leave behind a few hooks for future changes. Keep collecting metrics from function execution. If the profile of execution changes significantly, re-compile the function according to the new profile (possibly doing things like un-inlining).

This is perfectly aligned with a startup vs normal production use workflow. The only problems can appear if you have a continually changing pattern of execution through the same function.

Re: A Deep Introduction to JIT Compilers: JITs are not very Just-in-time

#65

Post by tsimionescu » Mon, Jul 06, 2020, 4:07 AM UTC

darksaints wrote: post #47

littlestymaar wrote: post #39

Earlier quoted context omitted.

> Same goes for the idea that garbage collection can be faster than manually managed memory. It's really workload dependent and depend a lot of the GC involved (a pretty dumb one like Python's or Go's won't get you anything performance wise), but a copying collector can achieve allocation way faster than a regular heap allocator (the allocation can be almost as cheap as allocating on the stack). If you can't avoid bo…

Yes, allocation can be incrementally be made cheaper with a copying GC. But you have to realize that it still comes at a cost, and allocation speed isn't always a bottleneck. You have a runtime process intercepting your code state, scanning and analyzing your heap and making a decision to keep, copy, or free. For anything larger than a trivially small object graph, that additional processing can easily overwhelm any…

Im very skeptical that the comparison really makes sense. Programs written in GC languages are so different in their allocation behavior from those written in manual memory languages that I don't think you can meaningfully compare them. And comparing regular C++ with Boehm GC is probably not that interesting, as Boehm is extremely limited by the semantics of C++.

And whichever way you look at it, the GC doesn't really have to outperform manual memory management to be extremely useful. It just has to be close enough, the correctness guarantees alone make it worth it as long as the performance difference is not too large, in many domains, not to mention the productivity boost and program readability.

Re: A Deep Introduction to JIT Compilers: JITs are not very Just-in-time

#66

Post by glitchc » Mon, Jul 06, 2020, 4:34 AM UTC

jakozaur wrote: post #2

Tl;dr: JIT is a compiler that optimuzes certain parts of code after interpreter see it as hot (e.g. was executed many time). Thank to runtime information it can occasionally exceed performance of statically compiled language.

I don’t believe JITs can ever beat AOT compilation. However, if it can achieve parity in the execution of the critical path, that is sufficient to call it a win.

Re: A Deep Introduction to JIT Compilers: JITs are not very Just-in-time

#67

Post by darksaints » Mon, Jul 06, 2020, 4:38 AM UTC

tsimionescu wrote: post #65

darksaints wrote: post #47

Earlier quoted context omitted.

Yes, allocation can be incrementally be made cheaper with a copying GC. But you have to realize that it still comes at a cost, and allocation speed isn't always a bottleneck. You have a runtime process intercepting your code state, scanning and analyzing your heap and making a decision to keep, copy, or free. For anything larger than a trivially small object graph, that additional processing can easily overwhelm any…

Im very skeptical that the comparison really makes sense. Programs written in GC languages are so different in their allocation behavior from those written in manual memory languages that I don't think you can meaningfully compare them. And comparing regular C++ with Boehm GC is probably not that interesting, as Boehm is extremely limited by the semantics of C++. And whichever way you look at it, the GC doesn't reall…

> And whichever way you look at it, the GC doesn't really have to outperform manual memory management to be extremely useful. It just has to be close enough, the correctness guarantees alone make it worth it as long as the performance difference is not too large, in many domains, not to mention the productivity boost and program readability.

I agree with all of this. But it is a very different argument than the theoretical argument that garbage collection can be faster than manual memory management.

Rust, for example, is manually managed or compiler managed, depending on how you look at it. But its semantics are extremely crude and naive: allocate on creation, free once an object leaves its current scope. There is no optimization of heap fragmentation, there is no optimization of large bulk frees...the compiler literally just inserts the malloc/free statements for you in a static location that you don't get to choose.

And despite the crudeness of its memory allocation behavior, it runs circles around garbage collected languages all day long. Even fast ones...I have done more than a handful of comparisons between Scala, Java, Go, SML (MLTon), OCaml, F#, and Rust...and Rust always comes out on top. That doesn't mean the performance advantages are worth the extra pain of dealing with a borrow checker...for most of my code, I default to Scala for all the reasons you've mentioned. But I don't delude myself into thinking it is faster.

Re: A Deep Introduction to JIT Compilers: JITs are not very Just-in-time

#68

Post by darksaints » Mon, Jul 06, 2020, 4:47 AM UTC

tsimionescu wrote: post #64

darksaints wrote: post #42

Earlier quoted context omitted.

Think of an use case where you have an API whose mode behavior is to do nothing, waiting for a call... but commonly gets calls that are very compute intense for short bursts of time. With the most common type of JITs, which profile once and compile once, I'm going to get code that is optimized for startup and initialization. If I have an "advanced" JIT, which is constantly deoptimizing and reoptimizing for whatever i…

The most commonly used JIT in the world is almost certainly the one in the OpenJDK, which is of the advanced kind. And it does not suffer from the problem you are talking about, because it only ever looks at code that is getting executed. Basically, it will do something like: interpret a function for a the first few thousand times it is executed, collecting execution metrics. After a threshold is reached, next time t…

HotSpot certainly does better than the vast majority of JITs out there, but it is definitely not perfect. If you have a JIT cache that is too small (really easy to do if you're not aware of the JIT settings or what they do), or your code does most of its heavy lifting in functions that are not called very often (because call frequency is the only heuristic it uses to prioritize optimization), you can easily trap yourself into horribly optimized executables that the JIT doesn't know its way out of.

Re: A Deep Introduction to JIT Compilers: JITs are not very Just-in-time

#69

Post by Reelin » Mon, Jul 06, 2020, 4:56 AM UTC

glitchc wrote: post #66

jakozaur wrote: post #2

Tl;dr: JIT is a compiler that optimuzes certain parts of code after interpreter see it as hot (e.g. was executed many time). Thank to runtime information it can occasionally exceed performance of statically compiled language.

I don’t believe JITs can ever beat AOT compilation. However, if it can achieve parity in the execution of the critical path, that is sufficient to call it a win.

> I don’t believe JITs can ever beat AOT compilation.

In certain cases [1] JIT can beat AOT by a decent margin because it has access to live information that isn't available at compile time. In theory you could AOT compile, profile, and recompile but in practice my understanding is that nothing beats doing it live.

Consider that if the workload suddenly changes in production and your heuristic violates an invariant as a result, you can deoptimize, reprofile, and recompile everything on the fly.

But wait, you say! Can't an AOT compiler do that? But no, the state space is almost certainly going to be too large by many orders of magnitude. So unless your AOT compiler effectively inserts a JIT compiler ...

[1] Sorry, no examples to hand. Nobody knows but I'm actually a dog. Don't trust me!

Re: A Deep Introduction to JIT Compilers: JITs are not very Just-in-time

#70

Post by jashmatthews » Mon, Jul 06, 2020, 5:21 AM UTC

darksaints wrote: post #67

tsimionescu wrote: post #65

Earlier quoted context omitted.

Im very skeptical that the comparison really makes sense. Programs written in GC languages are so different in their allocation behavior from those written in manual memory languages that I don't think you can meaningfully compare them. And comparing regular C++ with Boehm GC is probably not that interesting, as Boehm is extremely limited by the semantics of C++. And whichever way you look at it, the GC doesn't reall…

> And whichever way you look at it, the GC doesn't really have to outperform manual memory management to be extremely useful. It just has to be close enough, the correctness guarantees alone make it worth it as long as the performance difference is not too large, in many domains, not to mention the productivity boost and program readability. I agree with all of this. But it is a very different argument than the theor…

What you're really seeing the benefit of here isn't manual memory allocation but stack allocation. Hammering malloc/free in Rust is slower than using even a pretty simple GC which is why there are crates to do arena based allocation.

As a more concrete example splitting strings by copying in Rust and hammering malloc and memcpy is 4-5x slower than using slices in Go and letting the GC deal with keeping things alive.

You might say that's not a fair comparison but few businesses can tolerate the messing around that is writing zero copy Rust.

malloc implementations actually do optimize by delaying the work of free(1). They do a lot of complex stuff hidden behind those function calls to improve performance.