Travis Long had done something similar in 2022 at Mozilla.
https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...
10% of Firefox crashes are caused by bitflips
141–150 of 495 posts
Re: 10% of Firefox crashes are caused by bitflips
#142Very interesting. The Go toolchain has an (off by default) telemetry system. For Go 1.23, I added the runtime.SetCrashOutput function and used it to gather field reports containing stack traces for crashes in any running goroutine. Since we enabled it over a year ago in gopls, our LSP server, we have discovered hundreds of bugs. Even with only about 1 in 1000 users enabling telemetry, it has been an invaluable source…
You might consider adding the CPU temperature to the report, if there's a reasonable way to get it (haven't tried inside a VM). Then you could at least filter out extremely hot hardware.
Re: 10% of Firefox crashes are caused by bitflips
#143Earlier quoted context omitted.
It's not even ECC price/availability that bothers me so much, it's that getting CPUs and motherboards that support ECC is non-trivial outside of the server space. The whole consumer class ecosystem is kind of shitty. At least AMD allows consumer class CPUs to kinda sorta use ECC, unlike Intel's approach where only the prosumer/workstation stuff gets ECC.
I've been honestly amazed people actually buy stuff that's not "workstation" gear given IME how much more reliably and consistently it works, but I guess even a generation or two used can be expensive.
overblown? billions of users use consumer tier hardware just fine. i have servers at home with years of uptime without any ECC memory
Re: 10% of Firefox crashes are caused by bitflips
#144Earlier quoted context omitted.
Did you/he ever consider redundant allocation for high value content and hash checks for low value assets that are still important? I imagine the largest volume of game memory consumption is media assets which if corrupted would really matter, and the storage requirement for important content would be reasonably negligible?
I think the most reasonable take would be to just tell the users hardware is borked, they're going to have a bad outside the game too, and point them to one of the many guides around this topic. I don't think engineering effort should ever be put into handling literal bad hardware. But, the user would probably love you for letting them know how to fix all the crashing they have while they use their broken computer! T…
I think I sit in another camp. A lot of my engineering efforts are in working around bad hardware.
Better the user sees some lag due to state rebuild versus a crash.
Most consumers have what they have, and use what they have. Upgrading everything is now rare. If they got screwed, they'll remain screwed for a few years.
Re: 10% of Firefox crashes are caused by bitflips
#145> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.
Chromium has better handling for bitflip errors. Mostly due to the Discardable buffers they make such extensive use of.
The hardware bugs are there. They're just handled.
Re: 10% of Firefox crashes are caused by bitflips
#146> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.
Are people getting so many FF crashes? Mine rarely does. I leave it running, opening and closing tabs, for weeks on end.
How many DRM-heavy websites do you use? Widevine is a buggy thing.
Re: 10% of Firefox crashes are caused by bitflips
#147> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.
Are people getting so many FF crashes? Mine rarely does. I leave it running, opening and closing tabs, for weeks on end.
I run FF on Mac laptop, Windows/Linux laptop, and Windows desktop and can’t remember it crashing in years.
Re: 10% of Firefox crashes are caused by bitflips
#148I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…
Yup! I've read this decade ago... https://www.codeofhonor.com/blog/whose-bug-is-this-anyway
for people that dont know, www.codeofhonor.com is netcoyotes (the gp comment) blog, and there is some good reading to be had there
Re: 10% of Firefox crashes are caused by bitflips
#149Earlier quoted context omitted.
Did you/he ever consider redundant allocation for high value content and hash checks for low value assets that are still important? I imagine the largest volume of game memory consumption is media assets which if corrupted would really matter, and the storage requirement for important content would be reasonably negligible?
I think the most reasonable take would be to just tell the users hardware is borked, they're going to have a bad outside the game too, and point them to one of the many guides around this topic. I don't think engineering effort should ever be put into handling literal bad hardware. But, the user would probably love you for letting them know how to fix all the crashing they have while they use their broken computer! T…
I put engineering effort into handling bad hardware all the time because safety critical, :)
It significantly overlaps the engineering to gracefully handle non-hardware things like null pointers and forgetting to update one side of a communication interface.
80/20 rule, really. If you're thoughtful about how you build, you can get most of the benefits without doing the expensive stuff.
Re: 10% of Firefox crashes are caused by bitflips
#150Earlier quoted context omitted.
Everyone who has put serious effort into analyzing crash reports en mass has made similar discoveries that some portion of their crashes are best explained by faulty hardware. What percent that is mostly comes down to how stable your software is. The more bugs you have, the lower the portion that come from hardware. Firefox being at 10% from bad RAM just means that crashes due to FF bugs are somewhat uncommon but not…
IME, random bitflips is the engineer's way of saying "I'm sick and tired of root cause analysis" or "I have no fucking clue what the bug is." I, like others, remain skeptical about the claim.
“I have no data, but I’m sure those who do have data, and have spent a significant amount of time analyzing it, are wrong.”