Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

321–330 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#321

Post by j16sdiz »

crazygringo wrote: post #124
nimih wrote: post #111

Earlier quoted context omitted.

> Bold claim. I agree. Good thing he doesn't back up his claim with any sort of evidence or reasoned argument, or you'd look like a huge moron!

To be fair, he doesn't really: > And because it's a conservative heuristic we're underestimating the real number, it's probably going to be at least twice as much. The actual measurement is 5%. The 10% figure is entirely made up, with zero evidence or reasoned argument except a hand-wavy "conservative". Edit: actually, the claim is even less supported: > out of these ~25000 crashes have been detected as having a pote…

It depends on how the data are distributed.

I wouldn't be too surprised if that 5% all come from a few particular bad machine.

Re: 10% of Firefox crashes are caused by bitflips

#323

Post by SunnyNeon »

netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

How did you determine which of the causes it was?

Re: 10% of Firefox crashes are caused by bitflips

#324

Post by bell-cot »

Helmut10001 wrote: post #207
netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

I don't understand why ECC memory is not the norm these days. It is only slightly more expensive, but solves all these problems. Some consumer mainboards even support it already.

Talk to someone in consumer sales about customer priorities. A bit-cheaper computer? Or one which which is, in theory, more resilient against some rare random sort of problem which customers do not see as affecting them.

Re: 10% of Firefox crashes are caused by bitflips

#325

Post by AdamN »

haspok wrote: post #319
kleiba wrote: post #246

Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurpris…

The most frequent crashes I have with Firefox are when I type in a text area (such as this one right now, or on Reddit, for example). The longer the text I type is, the more probable it is that it's going to crash. Or maybe it doesn't crash, just grinds to such a slow pace that it is equivalent to a crash. My suspicion has always been some kind of a memory leak, but memory corruption also makes sense. Unfortunately,…

It could be a leak but it could also be an inefficient piece of logic in Firefox. One could imagine that on every keystroke Firefox is scanning the entire input text for typos or malicious inputs whereas Chrome might be scanning only the text before the cursor back until the first whitespace (since the other text is already known).

Re: 10% of Firefox crashes are caused by bitflips

#326

Post by chedabob »

versteegen wrote: post #314
PunchyHamster wrote: post #295

Earlier quoted context omitted.

> Several years later I learned that Dell computers had larger-than-reasonable analog component problems because Dell sourced the absolute cheapest stuff for their computers; I expect that was also a cause. Case in point: I was getting memory errors on my gaming machine, that persisted even after replacing the sticks. It caused windows bluesreen maybe once a month so I kinda lived with it as I couldn't afford to repl…

I'm surprised "faulty PSU" is not on GP's list of common problems. Almost every unstable computer I've ever experienced has been due to either a dying PSU (not an under-specced one) or dying power conversion capacitors on the motherboard.

Ye some of the weirdest issues I've fixed have been PSU related.

I had a PC come to me that would boot fine, but if you opened the CD drive it'd shut off instantly.

Re: 10% of Firefox crashes are caused by bitflips

#327

Post by matja »

jeffbee wrote: post #132
aforwardslash wrote: post #97

Earlier quoted context omitted.

ECC are traditionally slower, quite more complex, and they dont completely eliminate the problem (most memories correct 1 bit per word and detect 2 bits per word). They make sense when environmental factors such as flaky power, temperature or RF interference can be easily discarded - such as a server room. But yeah, I agree with you, as ECC solves like 99% of the cases.

ECC are "slower" because they are bought by smart people who expect their memory to load the stored value, rather than children who demand racing stripes on the DIMMs.

The actual RAM chips on a ECC DIMM are exactly the same as a non-ECC DIMM, there's just an extra 1/2/4 chips to extend to 72 bit words.

The main reason ECC RAM is slower is because it's not (by default) overclocked to the point of stability - the JEDEC standard speeds are used.

The other much smaller factors are:

* The tREFi parameter (refresh interval) is usually double the frequency on ECC RAM, so that it handles high-temperature operation. * Register chip buffers the command/address/control/clock signals, adding a clock of latency the every command (<1ns, much smaller than the typical memory latency you'd measure from the memory controller) * ECC calculation (AMD states 2 UMC cycles, <1ns).

Re: 10% of Firefox crashes are caused by bitflips

#328

Post by russdill »

aforwardslash wrote: post #97
Animats wrote: post #78

ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.

ECC are traditionally slower, quite more complex, and they dont completely eliminate the problem (most memories correct 1 bit per word and detect 2 bits per word). They make sense when environmental factors such as flaky power, temperature or RF interference can be easily discarded - such as a server room. But yeah, I agree with you, as ECC solves like 99% of the cases.

The amount of overhead a few bits of ECC has is basically a rounding error, and even then, the only time the hardware is really doing extra work is when bit errors occur and correction has to happen.

The main overhead is simply the extra RAM required to store the extra bits of ECC.

Re: 10% of Firefox crashes are caused by bitflips

#329

Post by matja »

SeanSullivan86 wrote: post #235

Hmm, can someone educate me here? Why don't bit flips ever seem to impact the results of calculations in settings like big-data analytics on AWS? Is it a difference between server hardware managed by knowledgeable people and random hardware thrown together by home PC builders?

You can only detect what you measure. Are these big-data analytics processes running multiple times to detect differences?

Re: 10% of Firefox crashes are caused by bitflips

#330

Post by Agentlien »

kasabali wrote: post #281
Agentlien wrote: post #222

Earlier quoted context omitted.

That's a really cool anecdote. The overclock makes sense. When we released Need For Speed (2015) I spent some time in our "war room", monitoring incoming crash reports and doing emergency patches for the worst issues. The vast majority of crashes came from two buckets: 1. PCs running below our minimum specs 2. Bugs in MSI Afterburner.

> Bugs in MSI Afterburner. Do you mean the OSD?

It seemed to be the monitoring side of it which caused a lot of crashes. It was apparently a very common issue in many games around that time.
Post reply on HN