Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

471–480 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#471

Post by rincebrain »

wging wrote: post #169
rincebrain wrote: post #39

Earlier quoted context omitted.

The simplest way to do this, what I believe memtest86 and friends do, is to write a fixed pattern over a region of memory and then read it back later and see if it changed; then you write patterns that require flipping the bits that you wrote before, and so on. Things like [1] will also tell you that something corrupted your memory, and if you see a nontrivial (e.g. lots of bits high and low) magic number that has on…

[4] looks like it's only a runner for the actual testing, which is a separate crate: https://github.com/mozilla/memtest (see: https://github.com/mozilla-firefox/firefox/blob/main/toolkit... , which points to a specific commit in that repo - turns out to be tip of main)

Yes.

I pointed to the runner because otherwise it seemed likely I'd get a comment insisting there's no proof it's used in Firefox.

Re: 10% of Firefox crashes are caused by bitflips

#472

Post by guenthert »

saati wrote: post #135
shevy-java wrote: post #51

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

I haven't seen a single firefox or chrome crash in months now, you should really stress-test your hardware.

And there's an app for that, aptly named stressapptest (originally developed by google). In the (now distant) past, I found it to be much more efficient (in terms of runtime until fault detected) and effective in finding memory related (RAM chips or memory controller) defects than memtest.

Re: 10% of Firefox crashes are caused by bitflips

#473

Post by agumonkey »

versteegen wrote: post #314
PunchyHamster wrote: post #295

Earlier quoted context omitted.

> Several years later I learned that Dell computers had larger-than-reasonable analog component problems because Dell sourced the absolute cheapest stuff for their computers; I expect that was also a cause. Case in point: I was getting memory errors on my gaming machine, that persisted even after replacing the sticks. It caused windows bluesreen maybe once a month so I kinda lived with it as I couldn't afford to repl…

I'm surprised "faulty PSU" is not on GP's list of common problems. Almost every unstable computer I've ever experienced has been due to either a dying PSU (not an under-specced one) or dying power conversion capacitors on the motherboard.

It's the powerhouse of the dell :p

Re: 10% of Firefox crashes are caused by bitflips

#474

Post by izacus »

grishka wrote: post #287
dpe82 wrote: post #152

Earlier quoted context omitted.

As a mobile dev at YouTube I'd periodically scroll through crash reports associated with code I owned and the long tail/non-clustered stuff usually just made absolutely no sense and I always assumed at least some of it was random bit flips, dodgy hardware, etc.

For the Mastodon Android app, I also sometimes see crashes that make no sense. For example, how about native crashes, on a thread that is created and run by the system, that only contains system libraries in its stack trace, and that never ran any of my code because the app doesn't contain any native libraries to begin with? Unfortunately I've never looked at crashes this way when I worked at VKontakte because there…

Framework, runtime, drivers and chips have bugs too. It's very easy to have some underlying component that corrupts your memory.

Re: 10% of Firefox crashes are caused by bitflips

#475

Post by jodrellblank »

MBCook wrote: post #433
jodrellblank wrote: post #430

Earlier quoted context omitted.

This is getting off-topic but I’m amazed by this ability to reach out to computers around the world as a sensor array and infer things we can’t easily find out in other ways. It’s in popular culture and HN comments most often as spyware and mass surveillance of people, and that’s a bit of a shame. GPS location and movement data is what gives Google maps its near-real-time view of traffic on all roads, and busy-ness o…

Doesn’t Google also use the phone accelerometer to try and spot earthquakes?

I don't know, but that's a good one. I wonder if they could do something like LIGO [1] which is an experiment of shining LASERS on mirrors 4km apart, to detect gravitational waves. Phone accelerometers don't have that kind of precision, but there are hundreds of millions of them and they are thousands of miles apart, is there possibly a signal among that noise?

[1] https://en.wikipedia.org/wiki/LIGO

Re: 10% of Firefox crashes are caused by bitflips

#476

Post by loeg »

rafaelmn wrote: post #468
loeg wrote: post #418

Earlier quoted context omitted.

And that consumer device should have ECC! That's the whole discussion here.

Why ? If your device is a thin client for web services/gaming the risk of bitflips/bad ram is a minor annoyance.

I disagree with your handwaving bitflips away as a minor annoyance. Consumers don't love software crashing, even if they don't have any data they care about.

Imagine ECC was free -- would you rather have free ECC and no bitflips, or no ECC and bitflips? It's hard to imagine choosing bitflips.

Re: 10% of Firefox crashes are caused by bitflips

#477

Post by hinkley »

akoboldfrying wrote: post #73
hinkley wrote: post #46

Earlier quoted context omitted.

Every so often when I'm doing refactoring work and my list of worries has decreased to the point I can start thinking of new things to worry about, I worry about how as we reduce the accidental complexity of code and condense the critical bytes of the working memory tighter and tighter, how we are leaning very hard on very few bytes and hoping none of them ever bitflip. I wonder sometimes if we shouldn't be doing lik…

Might be worth doing the kind of "manual ECC" you're describing for a small amount of high-importance data (e.g., the top few levels of a DB's B+ tree stored in memory), but I suspect the biggest win is just to use as little memory as possible, since the probability of being affected by memory corruption is roughly proportional to the amount you use.

Precautionary Principle is always about blast radius times probability. Condensing the state reduces the odds that the bit flip will be in your critical memory but increases the damage when it does. That tends to be a proportional amount so if it’s not a lateral move it’s at least a serpentine one.

Re: 10% of Firefox crashes are caused by bitflips

#478

Post by m000 »

funcDropShadow wrote: post #358
m000 wrote: post #316

Earlier quoted context omitted.

Maybe it's high time for some regulation? E.g. EU enforced mandatory USB-C charging from 2025, and pushes for ending production of combustion engine cars by 2035. Why not just make ECC RAM mandatory in new computers starting e.g. from 2030? AMD is already one step away from being compliant. So, it's not an outlandish requirement. And regulating will also force Intel to cut their BS, or risk losing the market.

OMG no. Politician have no business making technological decisions. They make it harder to innovate, i.e. to invent the next generation of ECC with a different name.

Reading this again, did you forget your trailing /s?

Re: 10% of Firefox crashes are caused by bitflips

#479

Post by akoboldfrying »

hinkley wrote: post #477
akoboldfrying wrote: post #73

Earlier quoted context omitted.

Might be worth doing the kind of "manual ECC" you're describing for a small amount of high-importance data (e.g., the top few levels of a DB's B+ tree stored in memory), but I suspect the biggest win is just to use as little memory as possible, since the probability of being affected by memory corruption is roughly proportional to the amount you use.

Precautionary Principle is always about blast radius times probability. Condensing the state reduces the odds that the bit flip will be in your critical memory but increases the damage when it does. That tends to be a proportional amount so if it’s not a lateral move it’s at least a serpentine one.

> but increases the damage when it does.

For this to be true, I think you would have to assume an "additive" model where each time corrupt memory is accessed it does some small amount of additional "damage". But for memory holding CPU instructions, I think it's more likely that the first time a corrupt byte is read, the program crashes.

Re: 10% of Firefox crashes are caused by bitflips

#480

Post by mafuy »

RealityVoid wrote: post #431
Agingcoder wrote: post #392

Earlier quoted context omitted.

Well my admins eventually believed me , so I’m fairly comfortable with what I said. We also had a few thousands of physical servers with about of terabyte of ram each. You are right : we did see repaired errors, but we also saw (indirectly, and after testing ) unrepaired ones

Ok, I am sure there is _some_ amount of unrepairable errors. But the initial discussion was that ECC ram makes it go away and your point that it doesn't. And the vast vast majority of the errors, according to my understanding and to the paper you pointed to, are repairable. About 1 out of 400 ish errors are non-repairable. That's a huge improvement! If you had ECC ram, the failures Firefox sees here would drop from 1…

You're thinking in terms of independent errors. I would think that this assumption is often not the case, so 3 errors right next to each other are comparatively likely to happen (far more than 3 individual errors). This would explain such 'strange' occurrences about ECC memory.
Post reply on HN