Laurence Tratt's paper Virtual machine warmup blows hot and cold [1] paper has been posted several times and never really discussed. It covers this problem for Java VMs and also presents a benchmarking methodology.
JavaScript Benchmarking Is a Mess
51–60 of 88 posts
Re: JavaScript Benchmarking Is a Mess
#52Re: JavaScript Benchmarking Is a Mess
#53Do the users care? I think they are used to waiting because they no longer know the speed of desktop applications.
it's fun to write "fast" js code and watch people's amazement. has hardware becomes cheaper and faster devs become lazier and careless.
it's all fun and games until your battery dies 3 hours too soon.
Re: JavaScript Benchmarking Is a Mess
#54Very strange take on "JIT introduce a lot of error into result". I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!? Only final-stage, fully-JIT-ted and profile-optimized code is what matter. Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to do…
JIT can be very unpredictable. I've seen cases with JVM of running the exact same benchmark in the same VM twice having the second run be 2x slower than the first, occurrences of having ran one benchmark before another making the latter 5x slower, and similar. Sure, if you make a 100% consistent environment of a VM running just the single microbenchmark you may get a consistent result on one system, but is a consiste…
How long did you run the benchmark if you got so large variation?
For simple methods I usually run the benchnarkes method 100k times, 10k is minimum for full JIT.
For large programs I have noticed the performance keeps getting better for the first 24 hours, after which I take a profiling dump.
Re: JavaScript Benchmarking Is a Mess
#55(I designed JavaScriptCore's optimizing JITs and its garbage collector and a bunch of the runtime. And I often benchmark stuff.) Here's my advice for how to run benchmarks and be happy with the results. - Any experiment you perform has the risk of producing an outcome that misleads you. You have to viscerally and spiritually accept this fact if you run any benchmarks. Don't rely on the outcome of a benchmark as if it…
> there's something like a 1/10 risk that you're fooling yourself.
You’re being generous or a touch ironic. It’s at least 1/10 and probably more like 1/5 on average and 1/3 for people who don’t take advice.
Beyond testing changes in a larger test fixture, I also find that sometimes multiplying the call count for the code under examination can help clear things up. Putting a loop in to run the offending code 10 times instead of once is a clearer signal. Of course it still may end up being a false signal.
I like a two phase approach, wheee you use a small scale benchmark while iterating on optimization ideas, with checking the larger context once you feel you’ve made progress, and again before you file a PR.
At the end of the day, eliminating accidental duplication of work is the most reliable form of improvement, and one that current and previous generation analysis tools don’t do well. Make your test cases deterministic and look at invocation counts to verify that you expect n calls of a certain shape to call the code in question exactly kn times. Then figure out why it’s mn instead. (This is why I say caching is the death of perf analysis. Once it’s added this signal disappears)
Re: JavaScript Benchmarking Is a Mess
#56kids dont recall when chrome was cheating left and right to be faster than firefox (after they were honestly for a couple months).
you'd have to run benchmarks for all sort of little thibgs because no browser would leave things be. If they thought one popular benchmark was using string+string it was all or nothing to optimize that, harming everything else. next week if that benchmark changed to string[].join... you get the idea. your code was all over the place in performance. Flying today, molasses next week... sometimes chrome and ff would switch the optimizations, so you'd serve string+string to one and array.join to the other. sigh.
Re: JavaScript Benchmarking Is a Mess
#57Re VM warmup, see https://tratt.net/laurie/blog/2022/more_evidence_for_problem... and the linked earlier research for some interesting discussion. Roughly, there is a belief when benchmarking that one can work around not having the most-optimised JIT-compiled version by running your benchmark a number of times and then throwing away the result before doing ‘real’ runs. But it turns out that: (a) sometimes the jit doe…
In general, this isn't even a JS problem, or a JIT problem. You have similar issues even in a lower-level language like C++: branch prediction, cache warming, heck, even power state transitions if you're using AVX512 instructions on an older CPU. Stop-the-world GC causing pauses? Variation in memory management exists in C, too -- malloc and free are not fixed-cost, especially under high churn. Benchmarks can be a use…
Running benchmarks on the old Intel MacBook a previous job gave me was like pulling teeth. Thermal throttling all the time. Anything less than at least a 2x speed up was just noise and I’d have to push my changes to CI/CD to test, which is how our build process sprouted a benchmark pass. And a grafana dashboard showing the trend lines over time.
My wheelhouse is making lots of 4-15% improvements and laptops are no good for those.
Re: JavaScript Benchmarking Is a Mess
#58For the love of god, please do not do this example: for (int i = 0; i Take your timing before and after the loop and divide by the count. Too much jitter otherwise. d8 and node have many options for benchmarking and if you really care, go command line. JSC is what is behind Bun so you can go that direction as well. And BTW: console.time et al does a bunch of stuff itself. You will get the JIT looking to optimize it a…
> and divide by the count Which gives an average rather than a time?
I usually do a
var innerCount = 2000; // should run about 2 seconds for (var i=0; iThat way I can both get enough precision form the millisecond resolution and run the whole thing enough times to get the best result without JIT/GC pauses. The result is usually very stable, even when benchmarking calls to database (running locally).
Re: JavaScript Benchmarking Is a Mess
#59That’s why i created mitata, it greatly improves on javascript (micro-)benchmarking tooling
it provides bunch of features to help avoiding jit optimization foot-guns during benchmarking and dips into more advanced stuff like hardware cpu counters to see what’s the end result of jit on cpu
Re: JavaScript Benchmarking Is a Mess
#60Earlier quoted context omitted.
I think that sort of thing is a bit different because you can have more control over it. If you’re serious about benchmarking or running particularly performance sensitive code, you’ll be able to get reasonably consistent benchmark results run-to-run, and you’ll have a big checklist of things like hugepages, pgo, tickless mode, writing code a specific way, and so on to get good and consistent performance. I think you…
The amount of control you have varies in a continuum between hand-written assembly to SQL queries. But there isn't really a difference of kind here, it's just a continuum. If there's anything unique about Javascript is that has an unusually high rate of "unpredictability" / "abstraction level". But again, it has pretty normal values of both of those, just the relation is away from the norm.
When we were upgrading to ES6 I was surprised/relieved to find that moving some leaf node code in the call graph to classes from prototypes did help. The common wisdom at the time was that classes were still relatively expensive. But they force strict, which we were using inconsistently, and they flatten the object representation (I discovered these issues in the heap dump, rather than the flame graph). Reducing memory pressure can overcome the cost of otherwise suboptimal code.
OpenTelemetry having not been invented yet, someone implemented server side HAR reports and the data collection on that was a substantial bottleneck. Particularly the original implementation.