Something I see many JS benchmarks struggle with is GC. Benchmarks run in tight loops with no GC in between, leading to results not representative of real world use. Rarely do they even run GC between the different test cases, so earlier cases build up GC pressure negatively impacting the later cases and invalidating all results.
JavaScript Benchmarking Is a Mess
81–88 of 88 posts
Re: JavaScript Benchmarking Is a Mess
#82Something I see many JS benchmarks struggle with is GC. Benchmarks run in tight loops with no GC in between, leading to results not representative of real world use. Rarely do they even run GC between the different test cases, so earlier cases build up GC pressure negatively impacting the later cases and invalidating all results.
yep, also thermal throttling.
had this discussion about GC pressure a bit ago: https://github.com/leeoniya/uDSV/issues/2
Re: JavaScript Benchmarking Is a Mess
#83(I designed JavaScriptCore's optimizing JITs and its garbage collector and a bunch of the runtime. And I often benchmark stuff.) Here's my advice for how to run benchmarks and be happy with the results. - Any experiment you perform has the risk of producing an outcome that misleads you. You have to viscerally and spiritually accept this fact if you run any benchmarks. Don't rely on the outcome of a benchmark as if it…
> there's something like a 1/10 risk that you're fooling yourself. You’re being generous or a touch ironic. It’s at least 1/10 and probably more like 1/5 on average and 1/3 for people who don’t take advice. Beyond testing changes in a larger test fixture, I also find that sometimes multiplying the call count for the code under examination can help clear things up. Putting a loop in to run the offending code 10 times…
The first half of the sentence you quoted is "even if you do everything right". What is the point of selectively quoting like that and then responding to something you know they didn't mean?
Re: JavaScript Benchmarking Is a Mess
#84Earlier quoted context omitted.
> Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to doesn't matter in grand scheme of things. This is true for servers but extremely not true for client-side GUI applications and web apps. Often, the entire process of [ user starts app > user performs a few tasks > user exits app ] can be done in a second. Often, the JIT n…
If it is done in literal second, why will you benchmark it? In such case you need "binary" benchmark: does user need to wait or not? You don't need some fancy graphics, percentiles, etc. And in such case your worse enemy is not JIT but variance of user's hardware, from old Atom netbook to high-end working station with tens of 5Ghz cores. Same for RAM and screen size.
> If it is done in literal second, why will you benchmark it?
The difference between one second and two seconds can be the difference between a happy user and an unhappy user.
> You don't need some fancy graphics, percentiles, etc.
You don't need those to tell you if your app is slow, you need them to tell you why your app is slow. The point of a profiler isn't to identity the existence of a performance problem. You should know you have a performance problem before you ever bother to start your profiler. The point is to give you enough information so that you can solve your performance problem.
> your worse enemy is not JIT but variance of user's hardware, from old Atom netbook to high-end working station with tens of 5Ghz cores. Same for RAM and screen size.
Yes, this are a real problem that client-side developers have to deal with. It's hard.
Re: JavaScript Benchmarking Is a Mess
#85Earlier quoted context omitted.
I'm not sure I understand the point of mentioning the language. If it was written in a language like c but still "shoved down your throat" would you still have qualms with it? Do you just like things written by corporate entities and those languages tend to be popular as they scale to larger teams well? Or do you dislike software that is too large to understand and optimized for the needs of larger teams? Because it…
The "need of larger teams" does not justify to delegate the core of the technical interfaces to a grotesquely and absurdely complex and gigantic computer language with its compiler (probably very few real life ones). This is accute lack of perspective, border-line fraud.
Not all problems are technical. Some are social. Velocity of a large team should absolutely be taken into account when designing software. Does the concept of an assembly line also upset you? Not every thing can be done by a single craftsperson working in isolation or in a small group.
I'm not suggesting where we ended up is ideal, but it has utility. I hope for a language reset akin to kotlin for the c++ community at some point to help resolve the complexity we've managed accumulate over decades. There is a decent amount of effort from various parts of the ecosystem to accomplish this.
Re: JavaScript Benchmarking Is a Mess
#86Earlier quoted context omitted.
Cross cutting concerns run into these sorts of problems. And they can sneak up on you because as you add these calls to your coding conventions, it’s incrementally added in new code and substantial edits to old, so what added a few tenths of a ms at the beginning may be tens of milliseconds a few years later. Someone put me on a trail of this sort last year and I managed to find about 75 ms of improvement (and anothe…
I think you might be responding to a different point than the one I made. The 1% I'm referring to is a 1% variation between subsequent runs of the same code. This is a measurement error, and it inhibits your ability to accurately compare the performance of two pieces of code that differ by a very small amount. Now, it reads like you' think I'm saying you shouldn't care about a method if it's only 1% of your runtime.…
> Now, it reads like you' think I'm saying you shouldn't care about a method if it's only 1% of your runtime. I definitely don't believe that.
Nor do I. I make a lot of progress down in the 3% code. You have to go file by file and module by module rather than hotspot by hotspot down here, and you have to be very good at refactoring and writing unit/pinning tests to get away with it. But the place I worked where I developed a lot of my theories I kept this up for over two years, making significant perf improvements every quarter before I had to start looking under previously visited rocks. It also taught me the importance of telegraphing competence to your customers. They will put up with a lot of problems if they trust you to solve them soon.
No I’m saying the jitter is a lot more than one percent, and rounding errors and externalities can be multiples of the jitter. Especially for leaf node functions - which are a fruitful target because it turns out your peers mostly don’t have opinions about “clever” optimizations if they’re in leaf node functions, in commits they could always roll back if they get sick of looking at them. Which is both a technical and a social problem.
One of the seminal cases for my leaving the conventional wisdom: I’d clocked a function call as 10% of an expensive route, via the profiler. I noticed that the call count was way off. A little more than twice what I would have guessed. Turned out two functions were making the call with the same input, but they weren’t far apart in the call tree, so I rearranged the code a bit to eliminate the second call, expecting the e2e time to be 5% faster. But it was repeatably 20% faster, which made no sense at all, except for memory pressure.
About five years later I saw the most egregious case of this during a teachable moment when I was trying to explain to coworkers that fixing dumb code >> caching. I eliminated a duplicate database request in sibling function calls, and we saw the entire response time drop by 10x instead of the 2-3x I was expecting.
Re: JavaScript Benchmarking Is a Mess
#87Earlier quoted context omitted.
> there's something like a 1/10 risk that you're fooling yourself. You’re being generous or a touch ironic. It’s at least 1/10 and probably more like 1/5 on average and 1/3 for people who don’t take advice. Beyond testing changes in a larger test fixture, I also find that sometimes multiplying the call count for the code under examination can help clear things up. Putting a loop in to run the offending code 10 times…
The first half of the sentence you quoted is "even if you do everything right". What is the point of selectively quoting like that and then responding to something you know they didn't mean?
With respect, the first half of the sentence sounds more like waffling than a clear warning to your peers. I bond with other engineers over how badly the industry as a whole handles the scientific method. Too many of us can’t test a thesis to the satisfaction of others. Hunches, speculation, and confidently incorrect. Every goddamn day.
Feynman said: The most important thing is not to fool yourself, and you’re the easiest person to fool.
That’s a lot more than 1/10, and he’s talking mostly to future scientists, not developers.
Re: JavaScript Benchmarking Is a Mess
#88Earlier quoted context omitted.
> and divide by the count Which gives an average rather than a time?
I usually do a var innerCount = 2000; // should run about 2 seconds for (var i=0; i That way I can both get enough precision form the millisecond resolution and run the whole thing enough times to get the best result without JIT/GC pauses. The result is usually very stable, even when benchmarking calls to database (running locally).
That’s great!
My point was to minimize stuff that will get JITed—like console functions. People don’t realize how much code is in there that is not native.