Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

JavaScript Benchmarking Is a Mess

byteofdev.com

Post reply on HN
Print view View original

71–80 of 88 posts

Re: JavaScript Benchmarking Is a Mess

#71

Post by austin-cheney »

There is a common sentiment I see there that I see regularly repeated in software. Here is my sarcastic take:

I hate measuring things because accuracy is hard. I wish I could just make up my own numbers to make myself feel better.

It is surprising to me how many developers cannot measure things, do so incorrectly, and then look for things to blame for their emotional turmoil.

Here is quick guide to solve for this:

1. Know what you are measuring and what its relevance is to your product. It is never about big or small because numerous small things make big things.

2. Measuring things means generating numbers and comparing those numbers against other numbers from a different but similar measure. The numbers are meaningless is there is no comparison.

3. If precision is important use the high performance tools provided by the browser and Node for measuring things. You can get greater than nanosecond precision and then account for the variance, that plus/minus range, in your results. If you are measuring real world usage and your numbers get smaller, due to performance refactoring, expect variance to increase. It’s ok, I promise.

4. Measure a whole bunch of different shit. The point of measuring things isn’t about speed. It’s about identifying bias. The only way to get faster is to know what’s really happening and just how off base your assumptions are.

5. Never ever trust performance indicators from people lacking objectivity. Expect to have your results challenged and be glad when they are. Rest on the strength of your evidence and ease of reproduction that you provide.

Re: JavaScript Benchmarking Is a Mess

#73

Post by skybrian »

Max-q wrote: post #31
skybrian wrote: post #19

Performance is inherently non-portable. In fact, ignoring performance differences is what enables portability. Not knowing what performance to expect is what allows you to build a website and expect it to run properly years later, on browsers that haven’t been released yet, running on future mobile phones that use chips that haven’t been designed yet, over a half-working WiFi connection in some cafe somewhere. Being…

Ignoring performance is what gives you slow code, costing you a lot if the code you write will be a success because you have to throw a lot more hardware at it. Think back to early Twitter that crashed and went down often hours each day. Most optimization will improve on all or some VMs. Most will not make it slower on others. If you write code that will be scaled up, optimization can save a lot of money, give better…

Sorry, I phrased that badly. By “ignoring performance,” I meant something more like “writing resilient code that can handle swings in performance.” For example, having generous deadlines where possible. Performance is fuzzy, but fast code that has lots of headroom will be more resilient than code that barely kees up.

Re: JavaScript Benchmarking Is a Mess

#74

Post by rgbrgb »

pizlonator wrote: post #39

(I designed JavaScriptCore's optimizing JITs and its garbage collector and a bunch of the runtime. And I often benchmark stuff.) Here's my advice for how to run benchmarks and be happy with the results. - Any experiment you perform has the risk of producing an outcome that misleads you. You have to viscerally and spiritually accept this fact if you run any benchmarks. Don't rely on the outcome of a benchmark as if it…

Love this. I have done a fair amount of UI performance optimization and agree with the end-to-end strategy.

For UX stuff, 2 steps I’d add if you're expecting a big improvement:

1) Ship some way of doing a sampled measurement in production before the optimization goes out. Networks and the spec of the client devices may be really important to the UX thing you're trying to improve. Likely user devices are different from your local benchmarking environment.

2) Try to tie it to a higher level metric (e.g. time on site, view count) that should move if the UI thing is faster. You probably don't just want it to be faster, you want the user to have an easier time doing their thing, so you want something that ties to that. At the very least this will build your intuition about your product and users.

Re: JavaScript Benchmarking Is a Mess

#75

Post by wavemode »

Max-q wrote: post #31
skybrian wrote: post #19

Performance is inherently non-portable. In fact, ignoring performance differences is what enables portability. Not knowing what performance to expect is what allows you to build a website and expect it to run properly years later, on browsers that haven’t been released yet, running on future mobile phones that use chips that haven’t been designed yet, over a half-working WiFi connection in some cafe somewhere. Being…

Ignoring performance is what gives you slow code, costing you a lot if the code you write will be a success because you have to throw a lot more hardware at it. Think back to early Twitter that crashed and went down often hours each day. Most optimization will improve on all or some VMs. Most will not make it slower on others. If you write code that will be scaled up, optimization can save a lot of money, give better…

To play devil's advocate: I don't think, in hindsight, Twitter would choose to have delayed their product's launch in order to engineer it for better performance. Given how wildly successful it became purely by being in the right place at the right time.

Re: JavaScript Benchmarking Is a Mess

#76

Post by mroche »

pizlonator wrote: post #39

(I designed JavaScriptCore's optimizing JITs and its garbage collector and a bunch of the runtime. And I often benchmark stuff.) Here's my advice for how to run benchmarks and be happy with the results. - Any experiment you perform has the risk of producing an outcome that misleads you. You have to viscerally and spiritually accept this fact if you run any benchmarks. Don't rely on the outcome of a benchmark as if it…

ThePrimeTime posted a livestream recording a few days ago where he and his guest dove into language comparison benchmarks. Even the first 10 minutes touches on things I hadn't thought of before beyond the obvious "these are not representative of real world workloads." It's an interesting discussion if you have the time.

https://www.youtube.com/watch?v=RrHGX1wwSYM

Re: JavaScript Benchmarking Is a Mess

#77

Post by hinkley »

ericyd wrote: post #25

Maybe I'm doing it wrong, but when I benchmark code, my goal is to compare two implementations of the same function and see which is faster. This article seems to be concerned with finding some absolute metric of performance, but to me that isn't what benchmarking is for. Performance will vary based on hardware and runtime which often aren't in your control. The limitations described in this article are interesting n…

It’s some of both because the same things that make benchmarking inaccurate make profiling less accurate. How do you know you should even be looking at that function in the first place?

You might get there eventually but a lot of people don’t persevere where perf is concerned. They give up early which leaves either a lot of perf on the table or room for peers to show them up.

Re: JavaScript Benchmarking Is a Mess

#78

Post by thayne »

pizlonator wrote: post #39

(I designed JavaScriptCore's optimizing JITs and its garbage collector and a bunch of the runtime. And I often benchmark stuff.) Here's my advice for how to run benchmarks and be happy with the results. - Any experiment you perform has the risk of producing an outcome that misleads you. You have to viscerally and spiritually accept this fact if you run any benchmarks. Don't rely on the outcome of a benchmark as if it…

I don't really disagree with anything you said, but having to run end to end tests for any benchmarking is far from ideal. For one thing, they are often slow,and to get reliable results, you have to run them multiple times, which makes it even slower. And that makes it more difficult, and expensive, to try a lot of things and iterate, and have a short feedback loop. IME writing good end to end tests is also just generally more difficult than writing unit tests of smaller benchmarking code.

Re: JavaScript Benchmarking Is a Mess

#80

Post by blacklion »

munificent wrote: post #70
blacklion wrote: post #5

Very strange take on "JIT introduce a lot of error into result". I'm from JVM/Java world, but it is JITted VM too, and in our world question is: why you want to benchmark interpreted code at all!? Only final-stage, fully-JIT-ted and profile-optimized code is what matter. Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to do…

> Short-lived interpreted / level-1 JITted code is not interesting at all from benchmarking perspective, because it will be compiled fast enough to doesn't matter in grand scheme of things. This is true for servers but extremely not true for client-side GUI applications and web apps. Often, the entire process of [ user starts app > user performs a few tasks > user exits app ] can be done in a second. Often, the JIT n…

If it is done in literal second, why will you benchmark it?

In such case you need "binary" benchmark: does user need to wait or not? You don't need some fancy graphics, percentiles, etc.

And in such case your worse enemy is not JIT but variance of user's hardware, from old Atom netbook to high-end working station with tens of 5Ghz cores. Same for RAM and screen size.

Post reply on HN