Benchmarking is a mess everywhere. Sure you can get some level of accuracy but reproducing any kind of benchmark results across machines is impossible. That's why perf people focus on things like CPU cycles, heap size, cache access etc instead of time. Even with multiple runs and averaged out results you can only get a surface level idea of how your code is actually performing.
JavaScript Benchmarking Is a Mess
11–20 of 88 posts
Re: JavaScript Benchmarking Is a Mess
#12Re VM warmup, see https://tratt.net/laurie/blog/2022/more_evidence_for_problem... and the linked earlier research for some interesting discussion. Roughly, there is a belief when benchmarking that one can work around not having the most-optimised JIT-compiled version by running your benchmark a number of times and then throwing away the result before doing ‘real’ runs. But it turns out that: (a) sometimes the jit doe…
I think Tratt’s work is great, but most of the effects that article highlights seem small enough that I think they’re most relevant to VM implementors measuring their own internal optimizations.
Iirc, the effects on long running benchmarks in that paper are usually < 1%, which is a big deal for runtime optimizations, but typically dwarfed by the differences between two methods you might measure.
Re: JavaScript Benchmarking Is a Mess
#13Very strange take on "JIT introduce a lot of error into result". I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!? Only final-stage, fully-JIT-ted and profile-optimized code is what matter. Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to do…
JIT can be very unpredictable. I've seen cases with JVM of running the exact same benchmark in the same VM twice having the second run be 2x slower than the first, occurrences of having ran one benchmark before another making the latter 5x slower, and similar.
Sure, if you make a 100% consistent environment of a VM running just the single microbenchmark you may get a consistent result on one system, but is a consistent result in any way meaningful if it may be a massive factor away from what you'd get in a real environment? And even then I've had cases of like 1.5x-2x differences for the exact same benchmark run-to-run.
Granted, this may be less of a benchmarking issue, more just a JIT performance issue, but it's nevertheless also a benchmarking issue.
Also, for JS, in browser specifically, pre-JIT performance is actually a pretty meaningful measurement, as each website load starts anew.
Re: JavaScript Benchmarking Is a Mess
#14Very strange take on "JIT introduce a lot of error into result". I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!? Only final-stage, fully-JIT-ted and profile-optimized code is what matter. Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to do…
Agreed, comparing functions in isolation can give you drastically different results from the real world, where your application can have vastly different memory access patterns.
Re: JavaScript Benchmarking Is a Mess
#15Re VM warmup, see https://tratt.net/laurie/blog/2022/more_evidence_for_problem... and the linked earlier research for some interesting discussion. Roughly, there is a belief when benchmarking that one can work around not having the most-optimised JIT-compiled version by running your benchmark a number of times and then throwing away the result before doing ‘real’ runs. But it turns out that: (a) sometimes the jit doe…
Benchmarking that isn't based on communicating the distribution of execution times is fundamentally wrong on almost any platform.
Re: JavaScript Benchmarking Is a Mess
#16If you use javascript, use a lean engine coded in a lean SDK, certainly not the c++ abominations in Big Tech web engines. Look at quickjs, and use your own very lean OS interfaces.
QuickJS is great for cases where you are more limited by startup and executable size than anything else but it tends to perform quite terribly ( https://bellard.org/quickjs/bench.html ) compared to V8 and anything else with JIT compilation. More code does not inherently mean worse performance.
But its SDK is not the c++ abominations like v8, and that alone is enough to choose quickjs or similar, since we all know here, that c++ (and similar, namely rust/java/etc) is a definitive nono (when it is not forced down our throat as a user, don't forget to thanks the guys doing that, usually well hidden behind internet...).
For performance, don't use javascript anyway...
That said, a much less worse middleground would be to have performance critical block written in assembly (RISC-V Now!) orchestrated by javascript.
Re: JavaScript Benchmarking Is a Mess
#17Re VM warmup, see https://tratt.net/laurie/blog/2022/more_evidence_for_problem... and the linked earlier research for some interesting discussion. Roughly, there is a belief when benchmarking that one can work around not having the most-optimised JIT-compiled version by running your benchmark a number of times and then throwing away the result before doing ‘real’ runs. But it turns out that: (a) sometimes the jit doe…
In general, this isn't even a JS problem, or a JIT problem. You have similar issues even in a lower-level language like C++: branch prediction, cache warming, heck, even power state transitions if you're using AVX512 instructions on an older CPU. Stop-the-world GC causing pauses? Variation in memory management exists in C, too -- malloc and free are not fixed-cost, especially under high churn. Benchmarks can be a use…
I think that sort of thing is a bit different because you can have more control over it. If you’re serious about benchmarking or running particularly performance sensitive code, you’ll be able to get reasonably consistent benchmark results run-to-run, and you’ll have a big checklist of things like hugepages, pgo, tickless mode, writing code a specific way, and so on to get good and consistent performance. I think you end up being less at the mercy of choices made by the VM.
Re: JavaScript Benchmarking Is a Mess
#18Very strange take on "JIT introduce a lot of error into result". I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!? Only final-stage, fully-JIT-ted and profile-optimized code is what matter. Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to do…
JIT can be very unpredictable. I've seen cases with JVM of running the exact same benchmark in the same VM twice having the second run be 2x slower than the first, occurrences of having ran one benchmark before another making the latter 5x slower, and similar. Sure, if you make a 100% consistent environment of a VM running just the single microbenchmark you may get a consistent result on one system, but is a consiste…
[deleted]
Re: JavaScript Benchmarking Is a Mess
#19Performance is inherently non-portable. In fact, ignoring performance differences is what enables portability.
Not knowing what performance to expect is what allows you to build a website and expect it to run properly years later, on browsers that haven’t been released yet, running on future mobile phones that use chips that haven’t been designed yet, over a half-working WiFi connection in some cafe somewhere.
Being ignorant of performance is what allows you to create Docker images that work on random servers in arbitrary datacenters, at the same time that perfect strangers are running their jobs and arbitrarily changing what hardware is available for your code to use.
It’s also what allows you to depend on a zillion packages written by others and available for free, and upgrade those packages without things horribly breaking due to performance differences, at least most of the time.
If you want fixed performance, you have to deploy on fixed, dedicated hardware, like video game consoles or embedded devices, and test on the same hardware that you’ll use in production. And then you drastically limit your audience. It’s sometimes useful, but it’s not what the web is about.
But faster is better than slower, so we try anyway. Understanding the performance of portable code is a messy business because it’s mostly not the code, it’s our assumptions about the environment.
We run tests that don’t generalize. For scientific studies, this is called the “external validity” problem. We’re often doing the equivalent of testing on mice and assuming the results are relevant for humans.
Re: JavaScript Benchmarking Is a Mess
#20Very strange take on "JIT introduce a lot of error into result". I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!? Only final-stage, fully-JIT-ted and profile-optimized code is what matter. Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to do…
Agreed, comparing functions in isolation can give you drastically different results from the real world, where your application can have vastly different memory access patterns.
Does anyone know how well the JIT/cache on the browser works eg. how useful it is to profile JIT'd vs non-JIT'd and what those different scenarios might represent in practice? For example is it just JIT-ing as the page loads/executes, or are there cached functions that persist across page loads, etc?