Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JavaScript Benchmarking Is a Mess

byteofdev.com

Post reply on HN
Print view View original

41–50 of 88 posts

Re: JavaScript Benchmarking Is a Mess

#41

Post by leeoniya »

pizlonator wrote: post #39

(I designed JavaScriptCore's optimizing JITs and its garbage collector and a bunch of the runtime. And I often benchmark stuff.) Here's my advice for how to run benchmarks and be happy with the results. - Any experiment you perform has the risk of producing an outcome that misleads you. You have to viscerally and spiritually accept this fact if you run any benchmarks. Don't rely on the outcome of a benchmark as if it…

great points! i do a lot of JS benchmarking + optimization and whole-program measurement is key. sometimes fixing one hotspot changes the whole profile, not just shifts the bottleneck to the next biggest thing in the original profile. GC behaves differently in different JS vms. sometimes if you benchmark something like CSV parsers which can stress the GC, Benchmark.js does a poor job by not letting the GC collect properly between cycles. there's a lengthy discussion about why i use a custom benchmark runner for this purpose [1]. i can recommend js-framework-benchmark [2] as a good example of one that is done well, also WebKit's speedometer [3].

[1] https://github.com/leeoniya/uDSV/issues/2

[2] https://github.com/krausest/js-framework-benchmark

[3] https://github.com/WebKit/Speedometer

Re: JavaScript Benchmarking Is a Mess

#42

Post by igouy »

sroussey wrote: post #36

For the love of god, please do not do this example: for (int i = 0; i Take your timing before and after the loop and divide by the count. Too much jitter otherwise. d8 and node have many options for benchmarking and if you really care, go command line. JSC is what is behind Bun so you can go that direction as well. And BTW: console.time et al does a bunch of stuff itself. You will get the JIT looking to optimize it a…

> and divide by the count

Which gives an average rather than a time?

Re: JavaScript Benchmarking Is a Mess

#43

Post by sylware »

surajrmal wrote: post #40
sylware wrote: post #16

Earlier quoted context omitted.

But its SDK is not the c++ abominations like v8, and that alone is enough to choose quickjs or similar, since we all know here, that c++ (and similar, namely rust/java/etc) is a definitive nono (when it is not forced down our throat as a user, don't forget to thanks the guys doing that, usually well hidden behind internet...). For performance, don't use javascript anyway... That said, a much less worse middleground w…

I'm not sure I understand the point of mentioning the language. If it was written in a language like c but still "shoved down your throat" would you still have qualms with it? Do you just like things written by corporate entities and those languages tend to be popular as they scale to larger teams well? Or do you dislike software that is too large to understand and optimized for the needs of larger teams? Because it…

The "need of larger teams" does not justify to delegate the core of the technical interfaces to a grotesquely and absurdely complex and gigantic computer language with its compiler (probably very few real life ones).

This is accute lack of perspective, border-line fraud.

Re: JavaScript Benchmarking Is a Mess

#44

Post by sylware »

ramon156 wrote: post #24
sylware wrote: post #16

Earlier quoted context omitted.

But its SDK is not the c++ abominations like v8, and that alone is enough to choose quickjs or similar, since we all know here, that c++ (and similar, namely rust/java/etc) is a definitive nono (when it is not forced down our throat as a user, don't forget to thanks the guys doing that, usually well hidden behind internet...). For performance, don't use javascript anyway... That said, a much less worse middleground w…

I could be reading your comment wrong, but what do you mean with "c++ is a definitive nono"? Also how is a complicated repository enough (reason) to choose quickjs?

quickjs or similar. Namely, small depending on a reasonable SDK (which does include the computer language).

Re: JavaScript Benchmarking Is a Mess

#45

Post by dan-robertson »

marcosdumay wrote: post #29
dan-robertson wrote: post #17

Earlier quoted context omitted.

I think that sort of thing is a bit different because you can have more control over it. If you’re serious about benchmarking or running particularly performance sensitive code, you’ll be able to get reasonably consistent benchmark results run-to-run, and you’ll have a big checklist of things like hugepages, pgo, tickless mode, writing code a specific way, and so on to get good and consistent performance. I think you…

The amount of control you have varies in a continuum between hand-written assembly to SQL queries. But there isn't really a difference of kind here, it's just a continuum. If there's anything unique about Javascript is that has an unusually high rate of "unpredictability" / "abstraction level". But again, it has pretty normal values of both of those, just the relation is away from the norm.

I’m quite confused by your comment. I think this subthread is about reasons one might see variance on modern hardware in your ‘most control’ end of the continuum.

The start of the thread was about some ways where VM warmup (and benchmarking) may behave differently from how many reasonably experienced people expect.

I claim it’s reasonably different because one cannot tame parts of the VM warmup issues whereas one can tame many of the sources of variability one sees in high-performance systems outside of VMs, eg by cutting out the OS/scheduler, by disabling power saving and properly cooling the chip, by using CAT to limit interference with the L3 cache, and so on.

Re: JavaScript Benchmarking Is a Mess

#46

Post by hyperpape »

ericyd wrote: post #25

Maybe I'm doing it wrong, but when I benchmark code, my goal is to compare two implementations of the same function and see which is faster. This article seems to be concerned with finding some absolute metric of performance, but to me that isn't what benchmarking is for. Performance will vary based on hardware and runtime which often aren't in your control. The limitations described in this article are interesting n…

The basic problem is that if the compiler handles your code in an unusual way in the benchmark, you haven't really measured the two implementations against each other, you've measured something different.

Dead code elimination is the most obvious way this happens, but you can also have issues where you give the branch predictor "help", or you can use a different number of implementations of a method so you get different inlining behavior (this can make a benchmark better or worse than reality), and many others.

As for runtime, if you're creating a library, you probably care at least a little bit about alternate runtimes, though you may well just target node/V8 (on the JVM, I've done limited benchmarking on runtimes other than HotSpot, though if any of my projects get more traction, I'd anticipate needing to do more).

Re: JavaScript Benchmarking Is a Mess

#48

Post by kmiller68 »

pizlonator wrote: post #39

(I designed JavaScriptCore's optimizing JITs and its garbage collector and a bunch of the runtime. And I often benchmark stuff.) Here's my advice for how to run benchmarks and be happy with the results. - Any experiment you perform has the risk of producing an outcome that misleads you. You have to viscerally and spiritually accept this fact if you run any benchmarks. Don't rely on the outcome of a benchmark as if it…

I completely agree with this advice. Micro-benchmarking can work well as long as you already have an understanding of what's happening behind the scenes. Without that it greatly increases the chance that you'll get information unrelated to how your code would perform in the real world. Even worse, I've found a lot of the performance micro-benchmarking websites can actually induce performance issues. Here's an example of a recent performance bug that appears to have been entirely driven by the website's harness. https://bugs.webkit.org/show_bug.cgi?id=283118

Re: JavaScript Benchmarking Is a Mess

#49

Post by egberts1 »

DJBunnies wrote: post #26

The whole JavaScript ecosystem is a mess.

As one who mapped the evolution of JavaScript and actually benchmark each of those iteration (company proprietary info), it Doesn't get anymore accurate that the OP's reiteration of the article's title.

I upvoted that.

Evolution chart:

https://egbert.net/blog/articles/javascript-jit-engines-time...

Re: JavaScript Benchmarking Is a Mess

#50

Post by marcosdumay »

dan-robertson wrote: post #45
marcosdumay wrote: post #29

Earlier quoted context omitted.

The amount of control you have varies in a continuum between hand-written assembly to SQL queries. But there isn't really a difference of kind here, it's just a continuum. If there's anything unique about Javascript is that has an unusually high rate of "unpredictability" / "abstraction level". But again, it has pretty normal values of both of those, just the relation is away from the norm.

I’m quite confused by your comment. I think this subthread is about reasons one might see variance on modern hardware in your ‘most control’ end of the continuum. The start of the thread was about some ways where VM warmup (and benchmarking) may behave differently from how many reasonably experienced people expect. I claim it’s reasonably different because one cannot tame parts of the VM warmup issues whereas one can…

> eg by cutting out the OS/scheduler, by disabling power saving and properly cooling the chip, by using CAT to limit interference with the L3 cache, and so on

That's not very different from targeting a single JS interpreter.

In fact, you get much more predictability by targeting a single JS interpreter than by trying to guess your hardware limitations... Except, of course if you target a single hardware specification.

Post reply on HN