Live data from Hacker News

Speedometer 3.0: A shared browser benchmark for web application responsiveness

browserbench.org

101–110 of 183 posts

Re: Speedometer 3.0: A shared browser benchmark for web application responsiveness

#101

Earlier quoted context omitted.

[flagged]

Is it though? https://wpt.fyi/interop-2024

Interop is only a tiny subset of the entire suite of WPT tests, and it only contains tests that all vendors agreed upon, so no browser will look bad in Interop.

If you look at the full WPT test suite [1], you'll see that Safari is by far the one failing the biggest number of tests, i.e. the most buggy browser.

The Safari team likes to use Interop to trick people into thinking Safari is as good as the others. It's just a PR play.

[1] https://wpt.fyi/results/?label=experimental&label=master&ali...

Re: Speedometer 3.0: A shared browser benchmark for web application responsiveness

#102
post #93

I tested this with Firefox stable release and Brave stable release, 3 runs on each. Same exact extensions across both. Highest scores across tests: Firefox: 6.34 +- 0.31 Brave: 11.3 +- 0.37 on Ryzen 9 7940HS + RTX 3060 mobile Which really sucks since I highly prefer Firefox but this past week I've been trying out Brave and I think its noticeably faster and smoother to me. Even with the reduced speed I'm still swayed…

The 'same' extensions on each browser control for your experience, but not for the browsers' performance:

The browsers have the same or very similar APIs for the extensions but that is just the interface; each browser executes the extension's instructions differently (a lot or a little - I don't know the browsers' code). The same extension will impact Brave's performance differently than it will impact Firefox's. In other words, the same extension is not, in this sense, the 'same' on each browser.

In this sense, an extension is part of the user experience, like a website. The Speedometer test suite doesn't include those extensions (I assume) and that is the experience the browsers are optimized for.

The parent's test doesn't represent that; it does represent their desired experience, of course.

Re: Speedometer 3.0: A shared browser benchmark for web application responsiveness

#103
post #23

Earlier quoted context omitted.

> You want more speed. On the other hand, premature optimization is the root of all evil. > Think of it as arbitrary units, which allows you to compare different version of browsers on the same machine. That's precisely the problem. It's arbitrary, meaningless. Without any physical units, I don't know what's good or bad, fast or slow. And why do the scores go from 0 to 140 when the web browsers are all getting approx…

The score goes from 0 to 140 so that there's some room for when the computers will get faster. When we started working on this, all browsers were maxed at 140, so the computation got changed.

I thought the front page goes to 140 just because it is modeled after actual GM dashboard speedometers produced ~1960-1990, sometimes having range 0-85 MPH, or 0-140km/h in metric markets.

Re: Speedometer 3.0: A shared browser benchmark for web application responsiveness

#104
post #43

Earlier quoted context omitted.

That’s not accurate. Firefox and all third party WebKit apps get the same JOT as Safari.

Right, but expecting the same behaviour from "Firefox" on iOS as on desktop is just not going to happen, since they have no control over the core engine. It's why, in general using iOS devices for cross-browser testing is pretty useless.

This is a fair point, though it is possible for app-level things that the browsers do to regress performance from the baseline pure engine level.

In this case, I think the 3 score must be either very old/low-end Android hardware or a measurement error. I don’t think any iOS browser gets 3.x scores, on even remotely modern hardware.

Re: Speedometer 3.0: A shared browser benchmark for web application responsiveness

#105
> The primary goal of Speedometer 3 is to reflect the real-world Web as much as possible, so that users benefit when a browser improves its score on the benchmark.

As with any other benchmark its results will be interpreted incorrectly and will have little effect on real world.

Google already has vast amounts of real-world data. The end result? "Oh, you should aim for a Largest Contentful Paint of 2.5 seconds or lower" (emphasis mine): https://blog.chromium.org/2020/05/the-science-behind-web-vit... Why? Because in real world the vast majority of sites is worse.

Browsers are already optimised beyond any reasonable expectation. Benchmarks like these focus on all the wrong things with little to no benefit to the actual performance of real-life web.

Make all benchmarks you want, but then Google's own Youtube will load 2.5 MB of CSS and 12 MB of Javascript to display a grid of images, and Google's own Lighthouse will scream at you for the hundreds of errors and warnings Youtube embed triggers.

Edit:

Optimise all you want, and run any benchmarks you want for the "real world", but performance inequality gap will still be there: https://infrequently.org/2024/01/performance-inequality-gap-...

Optimise all you want, and run any benchmarks you want for the "real world", but Lighthouse will warn you when you have over 800 DOM nodes, and will show an error for more than 1400 DOM nodes (which are laughably small numbers) for a reason: https://developer.chrome.com/docs/lighthouse/performance/dom...

Re: Speedometer 3.0: A shared browser benchmark for web application responsiveness

#106

Earlier quoted context omitted.

The score goes from 0 to 140 so that there's some room for when the computers will get faster. When we started working on this, all browsers were maxed at 140, so the computation got changed.

I thought the front page goes to 140 just because it is modeled after actual GM dashboard speedometers produced ~1960-1990, sometimes having range 0-85 MPH, or 0-140km/h in metric markets.

Nothing actually stops the score from going higher than 140, it will just max out the visual dashboard at that point. On Speedometer 2, Safari on M3 Macs ended up over 500. At scores that high it’s harder to have intuition, thus the changed scale of the new test.

Re: Speedometer 3.0: A shared browser benchmark for web application responsiveness

#107
In descending order....

MacBook Pro, M2 Pro, 16GB, plugged in, external display: Safari=31.2 Chrome=29.4

iPhone 12 mini, plugged in: Safari=19.4

HP Z2 mini (i7): Edge=15.9

Panasonic Toughbook CF19 (win 10): Edge=4.7 Chrome=5.6

Galaxy Tab S5e: Chrome=2.2

Oculus Quest 2: browser crashed

Tizen TV: displayed, wouldn't run

Nintendo 2DS: displayed, no css, wouldn't run

Re: Speedometer 3.0: A shared browser benchmark for web application responsiveness

#109
post #101

Earlier quoted context omitted.

Is it though? https://wpt.fyi/interop-2024

Interop is only a tiny subset of the entire suite of WPT tests, and it only contains tests that all vendors agreed upon, so no browser will look bad in Interop. If you look at the full WPT test suite [1], you'll see that Safari is by far the one failing the biggest number of tests, i.e. the most buggy browser. The Safari team likes to use Interop to trick people into thinking Safari is as good as the others. It's jus…

For a less biased result, use Stable: https://wpt.fyi/results/?label=master&label=stable&aligned

> If you look at the full WPT test suite [1], you'll see that Safari is by far the one failing the biggest number of tests, i.e. the most buggy browser.

In Safari's case, most WPT test fails mean "hasn't been implemented yet".

> Interop is only a tiny subset of the entire suite of WPT tests, and it only contains tests that all vendors agreed upon…

Exactly. If you're happy building "Works with Chrome" web apps, Safari is not for you.

Re: Speedometer 3.0: A shared browser benchmark for web application responsiveness

#110
post #25

On Firefox mobile I got a score of 3. So while Firefox is now super fast the performance of webapps might still be very bad on mobile. And I think this applies to all modern browsers: they are fast at rendering very slow webapps and websites.

4.7 brave on a pixel 7
Post reply on HN