Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

211–220 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#211

Post by colechristensen »

Helmut10001 wrote: post #207
netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

I don't understand why ECC memory is not the norm these days. It is only slightly more expensive, but solves all these problems. Some consumer mainboards even support it already.

Bit flips do not only happen inside RAM

Also, in a game, there is a tremendously large chance that any particular bit flip will have exactly 0 effect on anything. Sure you can detect them, but one pixel being wrong for 1/60th of a second isn't exactly ... concerning.

The chance for a bit flip to affect a critical path that is noticeable by the player is very low, and quite a bit lower if you design your game to react gracefully. There's a whole practice of writing code for radiation hardened environments that largely consists of strategies for recovering from an impossible to reach state.

Re: 10% of Firefox crashes are caused by bitflips

#212

Post by Helmut10001 »

colechristensen wrote: post #211
Helmut10001 wrote: post #207

Earlier quoted context omitted.

I don't understand why ECC memory is not the norm these days. It is only slightly more expensive, but solves all these problems. Some consumer mainboards even support it already.

Bit flips do not only happen inside RAM Also, in a game, there is a tremendously large chance that any particular bit flip will have exactly 0 effect on anything. Sure you can detect them, but one pixel being wrong for 1/60th of a second isn't exactly ... concerning. The chance for a bit flip to affect a critical path that is noticeable by the player is very low, and quite a bit lower if you design your game to react…

Interesting, I was not aware! Do you have a statistics for the bit flips in RAM %? My feeling would be its the majority of bit flips that happen, but I can be wrong.

Re: 10% of Firefox crashes are caused by bitflips

#213

Post by [deleted] »

colechristensen wrote: post #211
Helmut10001 wrote: post #207

Earlier quoted context omitted.

I don't understand why ECC memory is not the norm these days. It is only slightly more expensive, but solves all these problems. Some consumer mainboards even support it already.

Bit flips do not only happen inside RAM Also, in a game, there is a tremendously large chance that any particular bit flip will have exactly 0 effect on anything. Sure you can detect them, but one pixel being wrong for 1/60th of a second isn't exactly ... concerning. The chance for a bit flip to affect a critical path that is noticeable by the player is very low, and quite a bit lower if you design your game to react…

[deleted]

Re: 10% of Firefox crashes are caused by bitflips

#215

Post by ordu »

galangalalgol wrote: post #153
saati wrote: post #135

Earlier quoted context omitted.

I haven't seen a single firefox or chrome crash in months now, you should really stress-test your hardware.

I can't recall a single Firefox crash in at least a decade. What are people doing? I run ublock origin, nothing else. I do sometimes have Firefox mobile misbehave where it stops loading new pages and I jave to restart it, but open pages work normally as do all other operations, so not a crash exactly. Happens maybe once a month Edit: more context, I power cycle at least once a week on desktop and the version is typic…

Yeah. Lately even if I OOM my system, firefox doesn't crash so easily, individual tabs do.

Re: 10% of Firefox crashes are caused by bitflips

#216

Post by quesera »

stnvh wrote: post #14

Try running two instances of Firefox in parallel with different profiles, then do a normal quit / close operation on one after any use. Demons exist here.

Describe "demons"?

I run four Firefox instances simultaneously, most of the time. No issues to report.

Re: 10% of Firefox crashes are caused by bitflips

#218

Post by colechristensen »

Helmut10001 wrote: post #212
colechristensen wrote: post #211

Earlier quoted context omitted.

Bit flips do not only happen inside RAM Also, in a game, there is a tremendously large chance that any particular bit flip will have exactly 0 effect on anything. Sure you can detect them, but one pixel being wrong for 1/60th of a second isn't exactly ... concerning. The chance for a bit flip to affect a critical path that is noticeable by the player is very low, and quite a bit lower if you design your game to react…

Interesting, I was not aware! Do you have a statistics for the bit flips in RAM %? My feeling would be its the majority of bit flips that happen, but I can be wrong.

It would be quite hard to gather that data and would be highly dependent on hardware and source of bit flip.

But there's volatile and nonvolatile memory all over in a computer and anywhere data is in flight be it inside the CPU or in any wires, traces, or other chips along the data path can be subject to interference, cosmic rays, heat or voltage related errors, etc.

Re: 10% of Firefox crashes are caused by bitflips

#219

Post by ordu »

newscracker wrote: post #183

This is quite surprising to me, since I thought the percentage would be a lot lesser. But I don’t really know what the Firefox team does with crash reports and in making Firefox almost crash proof. I have been using it at work on Windows and for the last several years it always crashes on exit. I have religiously submitted every crash report. I even visit the “about:crashes” page to see if there are any unsubmitted o…

It is hard to judge, but a crash on exit seems to me a possible consequence of a damaged memory. Firefox frees all the resources and collects the garbage. I expect it to touch a lot of memory locations, and do something with values retrieved.

> this crash-on-exit happens due to many different causes, as seen in the crash reports

It points to the same direction: all these different causes are just symptoms, the root cause is hiding deeper, and it is triggered by the firefox stopping.

It is all is not a guarantee that the root cause is bitflips, but you can rule it out by testing your memory.

Re: 10% of Firefox crashes are caused by bitflips

#220

Post by rebelwebmaster »

newscracker wrote: post #183

This is quite surprising to me, since I thought the percentage would be a lot lesser. But I don’t really know what the Firefox team does with crash reports and in making Firefox almost crash proof. I have been using it at work on Windows and for the last several years it always crashes on exit. I have religiously submitted every crash report. I even visit the “about:crashes” page to see if there are any unsubmitted o…

Can you share a link to a crash report from about:crashes? Sounds like some kind of shutdown hang getting force-killed maybe?
Post reply on HN