Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

221–230 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#221

Post by netcoyote »

jiggawatts wrote: post #179
netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

Some multiplayer real-time strategy (RTS) games used deterministic fixed-point maths and incremental updates to keep the players in sync. Despite this, there would be the occasional random de-sync kicking someone out of a game, more than likely because of bit flips.

For RTS games I wish we could blame bit flips, but more typically it is uninitialized memory, incorrectly-not-reinitialized static variables, memory overwrites, use-after-free, non-deterministic functions (eg time), and pointer comparisons.

God I love C/C++. It’s like job security for engineers who fix bugs.

Re: 10% of Firefox crashes are caused by bitflips

#222

Post by Agentlien »

netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

That's a really cool anecdote. The overclock makes sense. When we released Need For Speed (2015) I spent some time in our "war room", monitoring incoming crash reports and doing emergency patches for the worst issues.

The vast majority of crashes came from two buckets:

1. PCs running below our minimum specs

2. Bugs in MSI Afterburner.

Re: 10% of Firefox crashes are caused by bitflips

#224

Post by KennyBlanken »

shevy-java wrote: post #51

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

"Software engineer thinks everyone's hardware is broken, couldn't possibly be bugs in his code" sums it up about right.

Re: 10% of Firefox crashes are caused by bitflips

#225

Post by ZiiS »

colechristensen wrote: post #218
Helmut10001 wrote: post #212

Earlier quoted context omitted.

Interesting, I was not aware! Do you have a statistics for the bit flips in RAM %? My feeling would be its the majority of bit flips that happen, but I can be wrong.

It would be quite hard to gather that data and would be highly dependent on hardware and source of bit flip. But there's volatile and nonvolatile memory all over in a computer and anywhere data is in flight be it inside the CPU or in any wires, traces, or other chips along the data path can be subject to interference, cosmic rays, heat or voltage related errors, etc.

It should be fairly easy to see statistically if ECC helps, people do run Firefox on it.

The number of bits in registers, busses, cache layers is very small compared to the number in RAM. Obviously they might be hotter or more likely to flip.

Re: 10% of Firefox crashes are caused by bitflips

#226

Post by squeaky-clean »

aforwardslash wrote: post #92
dheera wrote: post #87

Earlier quoted context omitted.

It says 10% of crashes If Firefox itself has so few bugs that it crashes very infrequently, it is not contradictory to what you are saying. I wouldn't be surprised if 99% of crashes in my "hello world" script are caused by bit flips.

Just updated with a comment. I see firefox crash routinely, so apparently our experiences are quite different :)

The last time I can recall Firefox crashing was when I was using Windows Vista. This definitely sounds like a problem with your system.

Re: 10% of Firefox crashes are caused by bitflips

#228

Post by justin66 »

Animats wrote: post #78

ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.

> ECC should have become standard around the time memories passed 1GB.

Ironically, that's around the time Intel started making it difficult to get ECC on desktop machines using their CPUs. The Pentium 3 and 440BX chipset, maxing out at 1GB, were probably the last combo where it pretty commonly worked with a normal desktop board and normal desktop processor.

Re: 10% of Firefox crashes are caused by bitflips

#229

Post by silon42 »

ordu wrote: post #215
galangalalgol wrote: post #153

Earlier quoted context omitted.

I can't recall a single Firefox crash in at least a decade. What are people doing? I run ublock origin, nothing else. I do sometimes have Firefox mobile misbehave where it stops loading new pages and I jave to restart it, but open pages work normally as do all other operations, so not a crash exactly. Happens maybe once a month Edit: more context, I power cycle at least once a week on desktop and the version is typic…

Yeah. Lately even if I OOM my system, firefox doesn't crash so easily, individual tabs do.

For me, OOM effectively crashes my system 90% of the time, usually caused by firefox (chromium too), if a website goes out of control (rarely it's caused by too many pages open, as tab discarding takes care of that).

Re: 10% of Firefox crashes are caused by bitflips

#230

Post by sgt »

hedora wrote: post #171
shevy-java wrote: post #51

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

I've had zero crashes in safari, ff or chrome in recent memory (except maybe OOMs). (Though I don't use Windows, so maybe that's part of the reason stuff just works?) Perhaps you're part of the group driving hardware crashes up to 10% and need to fix your machine.

I think most of it is just bad hardware, not specifically the RAM. Been using non-ECC desktop and laptop hardware for decades and I can't remember the machine crashing for .. I don't know, but a LONG time.
Post reply on HN