Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

411–420 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#411

Earlier quoted context omitted.

For safety critical systems, one strategy is to store at least two copies of important data and compare them regularly. If they don't match, you either try to recover somehow or go into a safe state, depending on the context.

At least three copies, so you can recover based on consensus.

In many cases the system is perfectly safe when it shuts off. Two is enough for that.

Re: 10% of Firefox crashes are caused by bitflips

#412
post #40

Earlier quoted context omitted.

That would tell you if there's a bitflip in your test, but not if there's a bitflip in normal program code causing a crash, no? IIUC GP's questions was how do they actually tell after a crash that that crash was caused by a bitflip.

The example I gave in there is of adding sentinel values in your data, so you can check the constants in your data structures later and go "oh, this is overwritten with garbage" versus "oh, this is one or two bits off". I would imagine plumbing things like that through most common structures is what was done there, though I haven't done the archaeology to find out, because Firefox is an enormous codebase to try and f…

This doesn't always protect against out-of-bounds writes. Although if these sentinel values are in read only memory mappings it probably gets pretty close. (Especially if you consider kernel memory corruption a "bitflip".)

Re: 10% of Firefox crashes are caused by bitflips

#413

I’ve also found that compiling large packages in GCC or similar tends to surface problems with the system’s RAM. Which probably means most typical software is resilient to a bit-flip; makes you wonder how many typos in actual documents might have been caused by bad R@M.

That's exactly how my bad RAM manifested itself. In fact, I was compiling Firefox, and gcc would get a segmentation fault at some random point during compilation. I'd have to clobber and restart the hour-long build. It was only when gcc started crashing while compiling other things that I even started considering the possibility of hardware failure. I'm a software developer, and based on what I produce myself, I just assume that all software is horribly buggy. ;-)

Re: 10% of Firefox crashes are caused by bitflips

#414
post #404

Earlier quoted context omitted.

Firefox has a long history of denying problems, blaming the user, and fixing the issue years later. It used to be memory usage, now it's crashing.

Did you actually read the posts that started this topic, or are you being an ass for no reason? Hint: No-one is claiming memory is to blame for 100% of the Firefox crashes. No-one is claiming it's 99% either.

Which part of my post was being an ass?

Sorry, but I experienced first hand Firefox's memory leaks not being taken seriously. This "bitflips" news is just released, but I fully expect anybody complaining about Firefox crashes to be met with low effort "It's your RAM," responses for the next few years now.

Re: 10% of Firefox crashes are caused by bitflips

#415
post #338

Earlier quoted context omitted.

At least three copies, so you can recover based on consensus.

“never go to sea with two chronometers, take one or three”

Seems like chronometers would be a case where two are better than one, because the mistakes are analog. If they don't exactly agree, just take the average. You'll have more error than if you were lucky enough to take the better chronometer, but less than if you had taken only the worse one. Minimizing the worst case is probably the best way to stay off the rocks.

Re: 10% of Firefox crashes are caused by bitflips

#416
I might be too late to this thread to get an answer but I do wonder how much of those bitflips are due to rowhammer-style attacks. Firefox runs trillions of lines of untrusted code a day with a non-insignificant part that is of malicious intent. I wouldn’t be shocked if some of those “analog” crashes are due to that.

Re: 10% of Firefox crashes are caused by bitflips

#417
post #7

The next logical step would be to somehow inform users so they could take action to replace the bad memory. I realize this is a challenge given the anonymized nature of the crash data, but I might be willing to trade some anonymity in exchange for stability.

I think the firefox crash reporter does now? It does a limited memory scan and reports problems it finds. No privacy violations required.

That's different from what you're suggesting, because you're right that the crash reports are analyzed with heuristics to guess at memory corruption. Aside from the privacy implications, though, I think that would have too many false alarms. A single bit flip is usually going to be an out of bounds write, not bad RAM.

Re: 10% of Firefox crashes are caused by bitflips

#418

Earlier quoted context omitted.

Compilers and test suits do scale (at least for C/C++ and Rust, which is what I work with). But I think the parent comment referred to consumer applications: games, word processing, light browsing, ... (Though games these days scale better than they used to, but only up to a to a point.) I find that most tools I write for my own use can be made to scale with cores, or run so fast that the overhead of starting threads…

But those tools aren't really compute bound anyway - you're not buying a workstation to do them, you're getting a consumer laptop or a tablet.

And that consumer device should have ECC! That's the whole discussion here.

Re: 10% of Firefox crashes are caused by bitflips

#419

Earlier quoted context omitted.

Very few applications scale with cores. For the vast majority of people single core performance is all they care about, it's also cheaper. They don't need or want workstation gear.

> Very few applications scale with cores You mean like compilers and test suites ? Very few professional workloads don't parallelize well these days.

Test suites often don't scale, actually. Unit tests usually run single-threaded by default, and also relatively often have side effects on the system that mean they're unsafe to run in parallel. (Sure, sure, you could definitely argue the latter thing is a skill issue.)
Post reply on HN