Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)

It would be better if VM comparisons included JSC rather than, or in addition to, V8. JSC tends to outperform V8 so if you find a pathology in V8 it’s just not so surprising. It would be more interesting if you found a pathology in JSC.

I think that the use of small benchmarks obscures what’s going on. The VM is trying to win in the average. It’s like a professional gambler. Observing that the VM did something dumb for a program is like observing that a professional gambler lost a bet. That’s not interesting. In a game of chance, even a really great strategy will have its outliers.

I think that to understand the quality of a VM you have to throw millions of lines of code at it and see if the optimizing JIT can consistently produced speedups or at least produced speedups more often than not using some aggregate metric. As someone who studies the behavior of JSC on million line code bases, I can tell you that a pretty good outcome is if only a small number of functions experience an “upside down” effect from optimization and ends up running slower over time.

Finally, the whole search for a methodology to pinpoint warmup is broken. It’s pure brain damage. VMs need to be fast even for small programs that don’t have a chance to warmup. Startup time is absolutely important. So it’s a methodological antipattern to even try to find the warmup.

The questions worth asking are:

- for some program, how long does it take to run that program. Start to finish. No ignoring warmup.

- how long does it take to run some very long program or the average running time of a small program averaged over many iterations

- some percentile of behavior, like the 99th, to get an average of the janky behavior.

Ideally you measure all of those things and include both short running and long running programs.

This tells you how good a VM is.

If you’re doing math or methodology to identify the warmup point then you’re effectively biasing your experiment to forgive VMs for bad behavior so long as that bad behavior happens early. Nothing could be sillier. Users care about the perf of their VMs at startup not just in steady state.

Anyway, that’s the way I like to do optimizations in JSC.

Some VMs are a lot better than others at startup.

Some programs don’t “run long enough” in the way the VM needs even when users run those programs.

Not measuring VM startup time has a long tradition in papers. It was the Original Sin (TM).

Totally true. JITs are just about making the dynamic languages perform better than if they were interpreted. If you want native perf you need a statically typed and ahead of time compiled language.

It occurs to me that, in practical terms, the "steady state" of performance with the increasingly large blobs of JavaScript we find littered throughout the web tends to impact the user more than other JITs. As the user navigates from page to page, the VM is reset, a fresh set of minified blobs is downloaded and JITed, and a core or two is pinned to do so. That translates pretty directly to a hotter phone, less battery life, and a frustrated user. Sure, when your JVM startup is 0.1% of the runtime of your program, it's not as big of a deal, but when it's more like 20% of the time a user spends using the program (their web browser), it's a lot worse, and has slim potential for improvement.