Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

61–70 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#61
post #50

I was running my PC with bad memory for a few weeks last year. Firefox crashed a LOT, way more than any other application I used during that time, so I've probably contributed a decent amount to these numbers...

It could be that firefox is written inefficiently though.

Or so efficiently that every bit counts and plays a vital role! Even a single bit off and the thing derails…

Re: 10% of Firefox crashes are caused by bitflips

#62
post #11

>> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! I find this impossible to believe. If this were so all devs for apps, games, etc... would be talking about this but since this is the first time I'm hearing about this I'm seriously doubting this. >> This is a bit skewed because users with flaky hardware will crash more often than users with fun…

Computers today have many GB of RAM, and programs that use it. The more RAM you have, the higher the probabilty that there will be some bad bits. And the more RAM a program uses, the more likely it will be using some that is bad. Same phenomenon with huge hard drives.

And most the time a bit flips it means that there's a wonky pixel somewhere in a photo, texture or video that you'd never even notice.

A bit flip actually needs to be pretty "lucky" to result in a crash.

Re: 10% of Firefox crashes are caused by bitflips

#64

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

>Sometimes I'm amazed that computers even work at all!

Funny you say this, because for a good while I was running OC'd RAM

I didn't see any instability, but Event Viewer was a bloodbath - reducing the speed a few notches stopped the entries (iirc 3800MHz down to 3600)

Re: 10% of Firefox crashes are caused by bitflips

#66
>That fancy ARM-based MacBook with RAM soldered on the CPU package? We've got plenty of crashes from those, good luck replacing that RAM without super-specialized equipment and an extraordinarily talented technician doing the job.

CPU caches and registers - how exactly are they different from a RAM on a SoC in this regard?

Re: 10% of Firefox crashes are caused by bitflips

#68

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

Maybe if Firefox tabs weren't such a memory hog it would be only 0.005% !

Re: 10% of Firefox crashes are caused by bitflips

#69

>That fancy ARM-based MacBook with RAM soldered on the CPU package? We've got plenty of crashes from those, good luck replacing that RAM without super-specialized equipment and an extraordinarily talented technician doing the job. CPU caches and registers - how exactly are they different from a RAM on a SoC in this regard?

For one thing, static vs dynamic RAM. Static RAM (which is what's used for your typical CPU cache) is implemented with flip-flops and doesn't need to be refreshed, reads aren't destructive like DRAM, etc.

Re: 10% of Firefox crashes are caused by bitflips

#70

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

> Bold claim. From my gut feeling this must be incorrect

RAM flips are common. This kind of thing is old and has likely gotten worse.

IBM had data on this. DEC had data on this. Amazon/Google/Microsoft almost certainly had data on this. Anybody who runs a fleet of computers gets data on this, and it is always eye opening how common it is.

ZFS is really good at spotting RAM flips.

Post reply on HN