Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

251–260 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#251

Post by qznc »

colinb wrote: post #239
colechristensen wrote: post #211

Earlier quoted context omitted.

Bit flips do not only happen inside RAM Also, in a game, there is a tremendously large chance that any particular bit flip will have exactly 0 effect on anything. Sure you can detect them, but one pixel being wrong for 1/60th of a second isn't exactly ... concerning. The chance for a bit flip to affect a critical path that is noticeable by the player is very low, and quite a bit lower if you design your game to react…

> code for radiation hardened environments I’m aware of code that detects bit flips via unreasonable value detection (“this counter cannot be this high so quickly”). What else is there?

The simplest one is a watchdog: If something stops with regular notifications, then restart stuff.

Re: 10% of Firefox crashes are caused by bitflips

#252

Post by rurban »

netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

I hate HW soo much. To revise the biggest problems in computing, beside out of tokens: HW bugs

Re: 10% of Firefox crashes are caused by bitflips

#253

Post by rockdoe »

d--b wrote: post #237

Does anyone know how they can detect hardware defects like this? This sounds like an incredibly hard problem. And I don’t see how they can do this without impacting performance significantly.

If the crash is isolated (no other reports) and flipping one bit in the crashing pointer value would make the pointer valid, it's assumed to be a bitflip. This obviously will only catch a minor portion of bitflips, i.e. any image or video data with bitflips wouldn't crash.

From what he's saying they run an actual memory test after a crash, too.

Re: 10% of Firefox crashes are caused by bitflips

#254

Post by Agingcoder »

kleiba wrote: post #246

Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurpris…

It depends on what you bitflip.

I once had a bitflip pattern causing lowercase ascii to turn into uppercase ascii in a case insensitive system. Everything was fine until it tried to uppercase numbers and things went wrong

The first time I had to deal with faulty ram ( more than 20y ago ), the bug would never trigger unless I used pretty much the whole dimm stick and put meaningful stuff in it etc in my case linking large executables , or untargzipping large source archives.

Flipping a pixel had no impact though

Re: 10% of Firefox crashes are caused by bitflips

#255

Post by gmueckl »

colinb wrote: post #239
colechristensen wrote: post #211

Earlier quoted context omitted.

Bit flips do not only happen inside RAM Also, in a game, there is a tremendously large chance that any particular bit flip will have exactly 0 effect on anything. Sure you can detect them, but one pixel being wrong for 1/60th of a second isn't exactly ... concerning. The chance for a bit flip to affect a critical path that is noticeable by the player is very low, and quite a bit lower if you design your game to react…

> code for radiation hardened environments I’m aware of code that detects bit flips via unreasonable value detection (“this counter cannot be this high so quickly”). What else is there?

For safety critical systems, one strategy is to store at least two copies of important data and compare them regularly. If they don't match, you either try to recover somehow or go into a safe state, depending on the context.

Re: 10% of Firefox crashes are caused by bitflips

#256

Post by rockdoe »

vlovich123 wrote: post #203
LeifCarrotson wrote: post #103

Earlier quoted context omitted.

Broken software causes null pointer references and similar logic errors. It would be extremely unusual to have an inadvertent ptr ^= (1 that got inserted in the code by accident. That's just not the way that we write software.

Except no one is claiming the bit flip is the pointer vs the data being pointed to or a non pointer value. Given how we write software there’s a lot more bits not in pointer values that still end up “contributing “ to a pointer value. Eg some offset field that’s added to a pointer has a bit flip, the resulting pointer also has a bit flip. But the offset field could have accidentally had a mask applied or a bit set ac…

I think that if you hit the crash in the same line of code many times, you can safely assume it's your own bug and not a memory issue.

If it's only hit once by a random person, memory starts being more likely.

(Unless that LOC is scanning memory or smth)

Re: 10% of Firefox crashes are caused by bitflips

#258

Post by gmueckl »

qznc wrote: post #251
colinb wrote: post #239

Earlier quoted context omitted.

> code for radiation hardened environments I’m aware of code that detects bit flips via unreasonable value detection (“this counter cannot be this high so quickly”). What else is there?

The simplest one is a watchdog: If something stops with regular notifications, then restart stuff.

A watchdog guards against unresponsive software. It doesn't protect against bad data directly. Not all bad data makes a system freeze.

Re: 10% of Firefox crashes are caused by bitflips

#259

Post by INTPenis »

That's super interesting because I remember Linus Torvalds saying he requires ECC RAM in his computers, because he got tired of weird issues that were resolved by a reboot.

But non-ECC is fine for most of us mortals gaming and streaming.

I would expect pro gamers to opt for ECC though.

Post reply on HN