Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

401–410 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#401

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

There's a very good chance your system also does not have flaky memory. Most don't. You're not contradicting the post.

Re: 10% of Firefox crashes are caused by bitflips

#402
post #319
post #246

Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurpris…

The most frequent crashes I have with Firefox are when I type in a text area (such as this one right now, or on Reddit, for example). The longer the text I type is, the more probable it is that it's going to crash. Or maybe it doesn't crash, just grinds to such a slow pace that it is equivalent to a crash. My suspicion has always been some kind of a memory leak, but memory corruption also makes sense. Unfortunately,…

I'm quite confident to say that millions of people use Firefox to comment on Reddit or similar sites every day, or write long posts, without seeing this problem.

Without knowing more about your configuration, it's hard to give advice, but definitely worth trying with a clean profile first.

If you don't report this problem upstream it will never get fixed, as obviously no-one else is seeing this. Firefox has a built-in profiler that you can use to report performance problems like this.

Re: 10% of Firefox crashes are caused by bitflips

#403

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

> Several years later I learned that Dell computers had larger-than-reasonable analog component problems because Dell sourced the absolute cheapest stuff for their computers; I expect that was also a cause.

Yikes. Dude, you're getting a Packard Bell.

Re: 10% of Firefox crashes are caused by bitflips

#404
post #246

Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurpris…

Firefox has a long history of denying problems, blaming the user, and fixing the issue years later. It used to be memory usage, now it's crashing.

Did you actually read the posts that started this topic, or are you being an ass for no reason?

Hint: No-one is claiming memory is to blame for 100% of the Firefox crashes. No-one is claiming it's 99% either.

Re: 10% of Firefox crashes are caused by bitflips

#405

Earlier quoted context omitted.

At least three copies, so you can recover based on consensus.

If your pieces of important data are very tiny, that's probably your best option. If they're hundreds of bytes or more, then two copies plus two hashes will do a better job.

A single hash should be enough.

Re: 10% of Firefox crashes are caused by bitflips

#406

Earlier quoted context omitted.

I can't recall a single Firefox crash in at least a decade. What are people doing? I run ublock origin, nothing else. I do sometimes have Firefox mobile misbehave where it stops loading new pages and I jave to restart it, but open pages work normally as do all other operations, so not a crash exactly. Happens maybe once a month Edit: more context, I power cycle at least once a week on desktop and the version is typic…

We have 5 computers running Firefox. One computer has regular Firefox crashes. I've done some memory testing that didn't detect anything wrong. I've tried all kinds of things software-wise but keep getting random crashes. I wonder if I should do a longer memory test, maybe some CPU stress testing at the same time...

If you want to dig into it, you can post a bunch of that computer's crash reports (navigate to about:crashes) on bugzilla: https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox&c...

Or you can view several of them and see if there's a common pattern in the "Signature" field. Firefox really should only be regularly crashing if: (1) there's a real bug and the thing that triggers it, (2) you're running out of memory, or (3) you have hardware.

I don't know what the odds of faulty hardware are for a randomly chosen user, but they're much higher for a randomly chosen user who is seeing regular crashes.

Re: 10% of Firefox crashes are caused by bitflips

#407

Earlier quoted context omitted.

I think that if you hit the crash in the same line of code many times, you can safely assume it's your own bug and not a memory issue. If it's only hit once by a random person, memory starts being more likely. (Unless that LOC is scanning memory or smth)

Deduplicating and identifying the source of a crash point is surprisingly hard, to the point that “it’s the only crash of its kind” could be a bug in your logic for linking issues. Also, in an unsafe language all bets are off. A memory clobber, UAF or race condition can generate quite strange and ephemeral crashes. Even if the majority of time it generates the “same” failure mode, it can still sporadically generate a…

Deduplicating and identifying the source of a crash point is surprisingly hard, to the point that “it’s the only crash of its kind” could be a bug in your logic for linking issues.

This is a bit vague to really reply to very specifically, but yes, this is hard. Which is why quite some people work in this area. It's rather valuable to do so at Firefox-scale.

Even if the majority of time it generates the “same” failure mode, it can still sporadically generate a rare execution trace.

This doesn't matter that much because the "same" failure mode already allows you to see the bug and fix it.

Re: 10% of Firefox crashes are caused by bitflips

#408
post #325
post #319

Earlier quoted context omitted.

The most frequent crashes I have with Firefox are when I type in a text area (such as this one right now, or on Reddit, for example). The longer the text I type is, the more probable it is that it's going to crash. Or maybe it doesn't crash, just grinds to such a slow pace that it is equivalent to a crash. My suspicion has always been some kind of a memory leak, but memory corruption also makes sense. Unfortunately,…

It could be a leak but it could also be an inefficient piece of logic in Firefox. One could imagine that on every keystroke Firefox is scanning the entire input text for typos or malicious inputs whereas Chrome might be scanning only the text before the cursor back until the first whitespace (since the other text is already known).

No.

Re: 10% of Firefox crashes are caused by bitflips

#409
post #301

>>> In the last week we received ~470000 crash reports, these do not represent all crashes because it's an opt-in system, the real number of crashes will be several times larger Having the number of unique machines would be great to see how skewed this estimate is.

To be fully accurate, it would also require tracking unique machines when collecting crash reports.

Re: 10% of Firefox crashes are caused by bitflips

#410

A 5 part thread where they say they're "now 100% positive" the crashes are from bitflips, yet not a single word is spent on how they're supposedly detecting bitflips other than just "we analyze memory"?

[flagged]

I don't think Firefox has the access permissions needed to read MCE status, and the vast majority of our users don't have ECC, let alone they're going to run memtest86(+) after a Firefox crash.

If they did, we wouldn't be having this discussion to begin with!

Post reply on HN