JavaScript Benchmarking Is a Mess
byteofdev.com
JavaScript Benchmarking Is a Mess
1–10 of 88 posts
Re: JavaScript Benchmarking Is a Mess
#2> Essentially, these differences just mean you should benchmark across all engines that you expect to run your code to ensure code that is fast in one isn’t slow in another.
In short, the JavaScript backend people now need to do what we JavaScript frontend people been doing since SPAs became a thing, run benchmarks across multiple engines instead of just one.
Re: JavaScript Benchmarking Is a Mess
#3If you use javascript, use a lean engine coded in a lean SDK, certainly not the c++ abominations in Big Tech web engines.
Look at quickjs, and use your own very lean OS interfaces.
Re: JavaScript Benchmarking Is a Mess
#4Re VM warmup, see https://tratt.net/laurie/blog/2022/more_evidence_for_problem... and the linked earlier research for some interesting discussion. Roughly, there is a belief when benchmarking that one can work around not having the most-optimised JIT-compiled version by running your benchmark a number of times and then throwing away the result before doing ‘real’ runs. But it turns out that:
(a) sometimes the jit doesn’t run
(b) sometimes it makes performance worse
(c) sometimes you don’t even get to a steady state with performance
(d) and obviously in the real world you may not end up with the same jitted version that you get in your benchmarks
Re: JavaScript Benchmarking Is a Mess
#5Very strange take on "JIT introduce a lot of error into result".
I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!?
Only final-stage, fully-JIT-ted and profile-optimized code is what matter.
Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to doesn't matter in grand scheme of things.
Re: JavaScript Benchmarking Is a Mess
#6Very strange take on "JIT introduce a lot of error into result". I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!? Only final-stage, fully-JIT-ted and profile-optimized code is what matter. Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to do…
> I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!?
Java gives you exceptional control over the JVM allowing you to create really good benchmark harnesses. That today is not the case with JavaScript and the proliferation of different runtimes makes that also harder. To the best of my knowledge there is no JMH equivalent for JavaScript today.
Re: JavaScript Benchmarking Is a Mess
#7If you use javascript, use a lean engine coded in a lean SDK, certainly not the c++ abominations in Big Tech web engines. Look at quickjs, and use your own very lean OS interfaces.
QuickJS is great for cases where you are more limited by startup and executable size than anything else but it tends to perform quite terribly (https://bellard.org/quickjs/bench.html) compared to V8 and anything else with JIT compilation.
More code does not inherently mean worse performance.
Re: JavaScript Benchmarking Is a Mess
#8> This effort, along with a move to prevent timing attacks, led to JavaScript engines intentionally making timing inaccurate, so hackers can’t get precise measurements of the current computers performance or how expensive a certain operation is.
The primary motivation for limiting timer resolution was the rise of speculative execution attacks (Spectre / Meltdown), where high-resolution timers are integral for differentiating between timings within the memory hierarchy.
https://github.com/google/security-research-pocs/tree/master...
If you look at when various browsers changed their timer resolutions, it's entirely a response to Spectre.
https://blog.mozilla.org/security/2018/01/03/mitigations-lan...
https://issues.chromium.org/issues/40556716 (SSCA -> "speculative side channel attacks")
Re: JavaScript Benchmarking Is a Mess
#9Re VM warmup, see https://tratt.net/laurie/blog/2022/more_evidence_for_problem... and the linked earlier research for some interesting discussion. Roughly, there is a belief when benchmarking that one can work around not having the most-optimised JIT-compiled version by running your benchmark a number of times and then throwing away the result before doing ‘real’ runs. But it turns out that: (a) sometimes the jit doe…
In general, this isn't even a JS problem, or a JIT problem. You have similar issues even in a lower-level language like C++: branch prediction, cache warming, heck, even power state transitions if you're using AVX512 instructions on an older CPU. Stop-the-world GC causing pauses? Variation in memory management exists in C, too -- malloc and free are not fixed-cost, especially under high churn.
Benchmarks can be a useful tool, but they should not be mistaken for real-world performance.
Re: JavaScript Benchmarking Is a Mess
#10For anyone interested in this subject, I’d recommend reading about JMH. The JVM isn’t 100% the same as JS VMs, but as a benchmarking environment it shares the same constraint of JIT compilation.
The right design is probably one that:
1) runs different tests in different forked processes, to avoid variance based on the order in which tests are run changing the JIT’s decisions.
2) runs tests for a long time (seconds or more per test) to ensure full JIT compilation and statistically meaningful results
Then you need to realize that your micro benchmarks give you information and help you understand, but the acid test is improving the performance of actual code.