Earlier quoted context omitted.
(I work on SpiderMonkey.) Benchmarking is hard. It is very easy to write a benchmark where improving your score does not improve real-world performance, and over time even a good benchmark will become less useful as the important improvements are all made. This V8 blog post about Octane is a good description of some of the issues: https://v8.dev/blog/retiring-octane Speedometer 3, in my experience, is the least bad b…
This doesn't say anything about what the scores mean. 21 is apparently better than 20, but how much better? You could say "1 better", tautologically, but how does that relate to the real world? Driving a car 1 mile per hour faster may be better, in a sense, but even if you drove 24 hours straight, it would only gain you 24 total miles, which is almost negligible on such a long trip. Nobody would be impressed by that…
> "The score is a rescaled version of inverse time" is the key here.
> If you run all the tests in half the time, your Speedometer score will double. If your score improves by 1%, it implies that you are 1% faster on the subtests.
> (There are probably some subtleties here because we're using the geometric mean to avoid putting too much weight on any individual subtest, but the rough intuition should still hold.)