Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JavaScript Benchmarking Is a Mess

byteofdev.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: JavaScript Benchmarking Is a Mess

#61

Post by hinkley »

hyperpape wrote: post #12
dan-robertson wrote: post #4

Re VM warmup, see https://tratt.net/laurie/blog/2022/more_evidence_for_problem... and the linked earlier research for some interesting discussion. Roughly, there is a belief when benchmarking that one can work around not having the most-optimised JIT-compiled version by running your benchmark a number of times and then throwing away the result before doing ‘real’ runs. But it turns out that: (a) sometimes the jit doe…

I think Tratt’s work is great, but most of the effects that article highlights seem small enough that I think they’re most relevant to VM implementors measuring their own internal optimizations. Iirc, the effects on long running benchmarks in that paper are usually < 1%, which is a big deal for runtime optimizations, but typically dwarfed by the differences between two methods you might measure.

Cross cutting concerns run into these sorts of problems. And they can sneak up on you because as you add these calls to your coding conventions, it’s incrementally added in new code and substantial edits to old, so what added a few tenths of a ms at the beginning may be tens of milliseconds a few years later. Someone put me on a trail of this sort last year and I managed to find about 75 ms of improvement (and another 50ms in stupid mistakes adjacent to the search).

And since I didn’t eliminate the logic I just halved the cost, that means we were spending about twice that much. But I did lower the slope of the regression line quite a lot, and I believe enough that new nodeJS versions improve response time faster than it was organically decaying. There were times it took EC2 instance type updates to see forward progress.

Re: JavaScript Benchmarking Is a Mess

#62

Post by wpollock »

ericyd wrote: post #25

Maybe I'm doing it wrong, but when I benchmark code, my goal is to compare two implementations of the same function and see which is faster. This article seems to be concerned with finding some absolute metric of performance, but to me that isn't what benchmarking is for. Performance will vary based on hardware and runtime which often aren't in your control. The limitations described in this article are interesting n…

You're not wrong, but there are cases where "absolute" performance matters. For example, when your app must meet a performance SLA.

Re: JavaScript Benchmarking Is a Mess

#63

Post by dzaima »

gmokki wrote: post #54
dzaima wrote: post #13

Earlier quoted context omitted.

JIT can be very unpredictable. I've seen cases with JVM of running the exact same benchmark in the same VM twice having the second run be 2x slower than the first, occurrences of having ran one benchmark before another making the latter 5x slower, and similar. Sure, if you make a 100% consistent environment of a VM running just the single microbenchmark you may get a consistent result on one system, but is a consiste…

How long did you run the benchmark if you got so large variation? For simple methods I usually run the benchnarkes method 100k times, 10k is minimum for full JIT. For large programs I have noticed the performance keeps getting better for the first 24 hours, after which I take a profiling dump.

Most of the simple benches I do are for ~1 second. The order-dependent things definitely were reproducible (something along the lines of rerunning resulting in some rare virtual method case finally being invoked enough times/with enough cases to heavily penalize the vastly more frequent case). And the case of very different results C2 deciding to compile the code differently (looking at the assembly was problematic as adding printassembly whatever skewed the case it took), and stayed stable for tens of seconds after the first ~second iirc (though, granted, it was preview jdk.incubator.vector code).

Re: JavaScript Benchmarking Is a Mess

#64

Post by igouy »

gmokki wrote: post #58
igouy wrote: post #42

Earlier quoted context omitted.

> and divide by the count Which gives an average rather than a time?

I usually do a var innerCount = 2000; // should run about 2 seconds for (var i=0; i That way I can both get enough precision form the millisecond resolution and run the whole thing enough times to get the best result without JIT/GC pauses. The result is usually very stable, even when benchmarking calls to database (running locally).

No interest in a more general tool?

https://github.com/sosy-lab/benchexec

Re: JavaScript Benchmarking Is a Mess

#65

Post by henning »

While there may be challenges, caring about frontend performance is still worth it. When I click the Create button in JIRA and start typing, the text field lags behind my typing. I use a 2019 MacBook Pro. Unforgivable. Whether one alternate implementation that lets me type normally is 10% faster than another or not or whatever may be harder to answer. If I measure how bad the UI is and it's actually 60x slower than vanilla JS rather than 70x because of measurement error, the app is still a piece of shit.

Re: JavaScript Benchmarking Is a Mess

#66

Post by spankalee »

My old team at Google created a tool to help do better browser benchmarking called Tachometer: https://github.com/google/tachometer

It tries to deal with the uncertainties of different browsers, JITs, GCs, CPU throttling, varying hardware, etc., several ways:

- Runs benchmarks round-robin to hopefully subject each implementation to varying CPU load and thermal properties evenly.

- It reports the confidence interval for an implementation, not the mean. Doesn't throw out outlier samples.

- For multiple implementations, compares the distributions of samples, de-emphasizing the mean

- For comparisons, reports an NxM difference table, showing how each impl compares to the other.

- Can auto-run until confidence intervals for different implementations no longer overlap, giving high confidence that there is an actual difference.

- Uses WebDriver to run benchmarks in multiple browsers, also round-robin, and compares results.

- Can manage npm dependencies, so you can run the same benchmark with different dependencies and see how different versions change the result.

Lit and Preact use Tachometer to tease out performance changes of PRs, even on unreliable GitHub Action hardware. We needed the advanced statistical comparisons exactly because certain things could be faster or slower in different JIT tiers, different browsers, or different code paths.

We wanted to be able to test changes that might have small but reliable overall perf impact, in the context of a non-micro-benchmark, and get reliable results.

Tachometer is browser-focused, but we made it before there were so many server runtimes. It'd be really interesting to make it run benchmarks against Node, Bun, Deno, etc. too.

Re: JavaScript Benchmarking Is a Mess

#67

Post by nemomarx »

spankalee wrote: post #66

My old team at Google created a tool to help do better browser benchmarking called Tachometer: https://github.com/google/tachometer It tries to deal with the uncertainties of different browsers, JITs, GCs, CPU throttling, varying hardware, etc., several ways: - Runs benchmarks round-robin to hopefully subject each implementation to varying CPU load and thermal properties evenly. - It reports the confidence interval f…

how relevant is browser benchmarking now that chrome owns most of the space?

Re: JavaScript Benchmarking Is a Mess

#68

Post by aardvark179 »

pizlonator wrote: post #39

(I designed JavaScriptCore's optimizing JITs and its garbage collector and a bunch of the runtime. And I often benchmark stuff.) Here's my advice for how to run benchmarks and be happy with the results. - Any experiment you perform has the risk of producing an outcome that misleads you. You have to viscerally and spiritually accept this fact if you run any benchmarks. Don't rely on the outcome of a benchmark as if it…

Excellent advice. It’s also very important to know what any micro benchmarks you do have are really measuring. I’ve seen enough that actually measured the time to setup or parse something because they dominated and wasn’t cached correctly. Conversely I’ve seen cases where the JIT correctly optimised away almost everything because there was a check on the final value.

Oh, and if each op takes under a nanosecond than your benchmark is almost certainly completely broken.

Re: JavaScript Benchmarking Is a Mess

#69

Post by hyperpape »

hinkley wrote: post #61
hyperpape wrote: post #12

Earlier quoted context omitted.

I think Tratt’s work is great, but most of the effects that article highlights seem small enough that I think they’re most relevant to VM implementors measuring their own internal optimizations. Iirc, the effects on long running benchmarks in that paper are usually < 1%, which is a big deal for runtime optimizations, but typically dwarfed by the differences between two methods you might measure.

Cross cutting concerns run into these sorts of problems. And they can sneak up on you because as you add these calls to your coding conventions, it’s incrementally added in new code and substantial edits to old, so what added a few tenths of a ms at the beginning may be tens of milliseconds a few years later. Someone put me on a trail of this sort last year and I managed to find about 75 ms of improvement (and anothe…

I think you might be responding to a different point than the one I made. The 1% I'm referring to is a 1% variation between subsequent runs of the same code. This is a measurement error, and it inhibits your ability to accurately compare the performance of two pieces of code that differ by a very small amount.

Now, it reads like you' think I'm saying you shouldn't care about a method if it's only 1% of your runtime. I definitely don't believe that. Sure, start with the big pieces, but once you've reached the point of diminishing returns, you're often left optimizing methods that individually are very small.

It sounds like you're describing a case where some method starts off taking However, you often still will be able to use micro-benchmarks to measure the difference between implementation A and implementation B, because odds are they differ not by 1% in their own performance, but 10% or 50% or something.

That's why I say that Tratt's work is great, but I think the variance it describes is a modest obstacle to most application developers, even if they're very performance minded.

Re: JavaScript Benchmarking Is a Mess

#70

Post by munificent »

blacklion wrote: post #5

Very strange take on "JIT introduce a lot of error into result". I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!? Only final-stage, fully-JIT-ted and profile-optimized code is what matter. Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to do…

> Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to doesn't matter in grand scheme of things.

This is true for servers but extremely not true for client-side GUI applications and web apps. Often, the entire process of [ user starts app > user performs a few tasks > user exits app ] can be done in a second. Often, the JIT never has a chance to warm up.

Post reply on HN