[dead]
JavaScript Benchmarking Is a Mess
21–30 of 88 posts
Re: JavaScript Benchmarking Is a Mess
#22Benchmarking is a mess everywhere . Sure you can get some level of accuracy but reproducing any kind of benchmark results across machines is impossible. That's why perf people focus on things like CPU cycles, heap size, cache access etc instead of time. Even with multiple runs and averaged out results you can only get a surface level idea of how your code is actually performing.
[deleted]
Re: JavaScript Benchmarking Is a Mess
#23Very strange take on "JIT introduce a lot of error into result". I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!? Only final-stage, fully-JIT-ted and profile-optimized code is what matter. Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to do…
Javascript code is often short lived and doesn't have enough time to wait for the JIT to watm up.
Re: JavaScript Benchmarking Is a Mess
#24Earlier quoted context omitted.
QuickJS is great for cases where you are more limited by startup and executable size than anything else but it tends to perform quite terribly ( https://bellard.org/quickjs/bench.html ) compared to V8 and anything else with JIT compilation. More code does not inherently mean worse performance.
But its SDK is not the c++ abominations like v8, and that alone is enough to choose quickjs or similar, since we all know here, that c++ (and similar, namely rust/java/etc) is a definitive nono (when it is not forced down our throat as a user, don't forget to thanks the guys doing that, usually well hidden behind internet...). For performance, don't use javascript anyway... That said, a much less worse middleground w…
I could be reading your comment wrong, but what do you mean with "c++ is a definitive nono"? Also how is a complicated repository enough (reason) to choose quickjs?
Re: JavaScript Benchmarking Is a Mess
#25Maybe I'm doing it wrong, but when I benchmark code, my goal is to compare two implementations of the same function and see which is faster. This article seems to be concerned with finding some absolute metric of performance, but to me that isn't what benchmarking is for. Performance will vary based on hardware and runtime which often aren't in your control. The limitations described in this article are interesting notes, but I don't see how they would stop me from getting a reasonable assessment of which implementation is faster for a single benchmark.
Re: JavaScript Benchmarking Is a Mess
#26The whole JavaScript ecosystem is a mess.
Re: JavaScript Benchmarking Is a Mess
#27Maybe I'm doing it wrong, but when I benchmark code, my goal is to compare two implementations of the same function and see which is faster. This article seems to be concerned with finding some absolute metric of performance, but to me that isn't what benchmarking is for. Performance will vary based on hardware and runtime which often aren't in your control. The limitations described in this article are interesting n…
Well, the issue is that micro benchmarking in JS is borderline useless.
You can have some function that iterates over something and benchmark two different implementations and draw conclusions that one is better than the other.
Then, in real world, when it's in the context of some other code, you just can't draw conclusions because different engines will optimize the very same paths differently in different contexts.
Also, your micro benchmark may tell you that A is faster than B...when it's a hot function that has been optimized due to being used frequently. But then you find that B which is used only few times and doesn't get optimized will run faster by default.
It is really not easy nor obvious to benchmark different implementations. Let alone the fact that you have differences across engines, browsers, devices and OSs (which will use different OS calls and compiler behaviors).
Re: JavaScript Benchmarking Is a Mess
#28Maybe I'm doing it wrong, but when I benchmark code, my goal is to compare two implementations of the same function and see which is faster. This article seems to be concerned with finding some absolute metric of performance, but to me that isn't what benchmarking is for. Performance will vary based on hardware and runtime which often aren't in your control. The limitations described in this article are interesting n…
Well, the issue is that micro benchmarking in JS is borderline useless. You can have some function that iterates over something and benchmark two different implementations and draw conclusions that one is better than the other. Then, in real world, when it's in the context of some other code, you just can't draw conclusions because different engines will optimize the very same paths differently in different contexts.…
I guess I've just never seen any alternative to microbenchmarking in the JS world. Do you know of any projects that do "macrobenchmarking" to a significant degree so I could see that approach?
Re: JavaScript Benchmarking Is a Mess
#29Earlier quoted context omitted.
In general, this isn't even a JS problem, or a JIT problem. You have similar issues even in a lower-level language like C++: branch prediction, cache warming, heck, even power state transitions if you're using AVX512 instructions on an older CPU. Stop-the-world GC causing pauses? Variation in memory management exists in C, too -- malloc and free are not fixed-cost, especially under high churn. Benchmarks can be a use…
I think that sort of thing is a bit different because you can have more control over it. If you’re serious about benchmarking or running particularly performance sensitive code, you’ll be able to get reasonably consistent benchmark results run-to-run, and you’ll have a big checklist of things like hugepages, pgo, tickless mode, writing code a specific way, and so on to get good and consistent performance. I think you…
The amount of control you have varies in a continuum between hand-written assembly to SQL queries. But there isn't really a difference of kind here, it's just a continuum.
If there's anything unique about Javascript is that has an unusually high rate of "unpredictability" / "abstraction level". But again, it has pretty normal values of both of those, just the relation is away from the norm.
Re: JavaScript Benchmarking Is a Mess
#30Maybe I'm doing it wrong, but when I benchmark code, my goal is to compare two implementations of the same function and see which is faster. This article seems to be concerned with finding some absolute metric of performance, but to me that isn't what benchmarking is for. Performance will vary based on hardware and runtime which often aren't in your control. The limitations described in this article are interesting n…
Isn't that more profiling than benchmarking?