Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

41–50 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#41

Post by rincebrain »

rendaw wrote: post #40
rincebrain wrote: post #39

Earlier quoted context omitted.

The simplest way to do this, what I believe memtest86 and friends do, is to write a fixed pattern over a region of memory and then read it back later and see if it changed; then you write patterns that require flipping the bits that you wrote before, and so on. Things like [1] will also tell you that something corrupted your memory, and if you see a nontrivial (e.g. lots of bits high and low) magic number that has on…

That would tell you if there's a bitflip in your test, but not if there's a bitflip in normal program code causing a crash, no? IIUC GP's questions was how do they actually tell after a crash that that crash was caused by a bitflip.

The example I gave in there is of adding sentinel values in your data, so you can check the constants in your data structures later and go "oh, this is overwritten with garbage" versus "oh, this is one or two bits off". I would imagine plumbing things like that through most common structures is what was done there, though I haven't done the archaeology to find out, because Firefox is an enormous codebase to try and find one person's commits from several years ago in.

Re: 10% of Firefox crashes are caused by bitflips

#42

Post by hexyl_C_gut »

thegrim33 wrote: post #3

A 5 part thread where they say they're "now 100% positive" the crashes are from bitflips, yet not a single word is spent on how they're supposedly detecting bitflips other than just "we analyze memory"?

It sounds like they don't know that the crashes are from bitflips but those crashes are from people with flaky memory which probably caused the crash?

Re: 10% of Firefox crashes are caused by bitflips

#43

Post by hinkley »

foresto wrote: post #29
vsgherzi wrote: post #8

is there a way to get the memory tester he mentioned? Is it open source? Once Ram goes bad is there a way or recovering it or is it toasted forever?

You can map known-bad memory regions to avoid using them. https://www.memtest86.com/blacklist-ram-badram-badmemorylist...

However if the third chip on your memory stick is properly broken, then the third bit out of every word of memory may get stuck high or low, and then the whole chip is absolutely worthless.

The most expensive memory failure I had was of this sort, and frustratingly came from accidentally unplugging the wrong computer.

After this I did buy some used memory from a recycling center that had the sorts of problems you described and was able to employ them by masking off the bad regions.

Re: 10% of Firefox crashes are caused by bitflips

#44

Post by Analemma_ »

netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

There's a famous Raymond Chen post about how a non-trivial percentage of the blue screen of death reports they were getting appeared to be caused by overclocking, sometimes from users who didn't realize they had been ripped off by the person who sold them the computer: https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35.... Must've been really frustrating.

Re: 10% of Firefox crashes are caused by bitflips

#45

Post by sieep »

adonovan wrote: post #25

Very interesting. The Go toolchain has an (off by default) telemetry system. For Go 1.23, I added the runtime.SetCrashOutput function and used it to gather field reports containing stack traces for crashes in any running goroutine. Since we enabled it over a year ago in gopls, our LSP server, we have discovered hundreds of bugs. Even with only about 1 in 1000 users enabling telemetry, it has been an invaluable source…

Ive been trying to push my boss towards more analytics/telemetry in production that focus on crashes, thanks for sharing.

Re: 10% of Firefox crashes are caused by bitflips

#46

Post by hinkley »

mrguyorama wrote: post #9

People I think are overindexing on this being about "Bad hardware". We have long known that single bit errors in RAM are basically "normal" in terms of modern computers. Google did this research in 2009 to quantify the number of error events in commodity DRAM https://static.googleusercontent.com/media/research.google.c... They found 25,000 to 70,000 errors per billion device hours per Mbit and more than 8% of DIMMs a…

Every so often when I'm doing refactoring work and my list of worries has decreased to the point I can start thinking of new things to worry about, I worry about how as we reduce the accidental complexity of code and condense the critical bytes of the working memory tighter and tighter, how we are leaning very hard on very few bytes and hoping none of them ever bitflip.

I wonder sometimes if we shouldn't be doing like NASA does and triple-storing values and comparing the calculations to see if they get the same results.

Re: 10% of Firefox crashes are caused by bitflips

#47

Post by SoftTalker »

NotGMan wrote: post #11

>> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! I find this impossible to believe. If this were so all devs for apps, games, etc... would be talking about this but since this is the first time I'm hearing about this I'm seriously doubting this. >> This is a bit skewed because users with flaky hardware will crash more often than users with fun…

Computers today have many GB of RAM, and programs that use it.

The more RAM you have, the higher the probabilty that there will be some bad bits. And the more RAM a program uses, the more likely it will be using some that is bad.

Same phenomenon with huge hard drives.

Re: 10% of Firefox crashes are caused by bitflips

#48

Post by SoftTalker »

shiroiuma wrote: post #31
mrguyorama wrote: post #9

People I think are overindexing on this being about "Bad hardware". We have long known that single bit errors in RAM are basically "normal" in terms of modern computers. Google did this research in 2009 to quantify the number of error events in commodity DRAM https://static.googleusercontent.com/media/research.google.c... They found 25,000 to 70,000 errors per billion device hours per Mbit and more than 8% of DIMMs a…

It'd be interesting to see how your experience would differ if you put it to sleep at night after switching to ECC RAM. Unfortunately, not that many consumer platforms make this possible or affordable.

Most computers running Firefox won't have ECC RAM.

Re: 10% of Firefox crashes are caused by bitflips

#49

Post by hinkley »

titaniumtravel wrote: post #19
nubinetwork wrote: post #18

Earlier quoted context omitted.

2% worldwide? https://gs.statcounter.com/browser-market-share Granted, they're probably just as accurate as netcraft. /shrug

The nuance here is of cause that there are a bunch of people using multiple browsers. Also I mean there are a lot of people using browsers on the world

If 10% of firefox users are also iOS users, which is not unlikely, then those people get double-counted. In my case I probably use my phone and tablet for at least 50% of my web traffic, not counting youtube, which also skews things.
Post reply on HN