Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

201–210 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#201

Earlier quoted context omitted.

I've been honestly amazed people actually buy stuff that's not "workstation" gear given IME how much more reliably and consistently it works, but I guess even a generation or two used can be expensive.

overblown? billions of users use consumer tier hardware just fine. i have servers at home with years of uptime without any ECC memory

But how much bit rot? You’ll never know.

Re: 10% of Firefox crashes are caused by bitflips

#202
I’ve also found that compiling large packages in GCC or similar tends to surface problems with the system’s RAM. Which probably means most typical software is resilient to a bit-flip; makes you wonder how many typos in actual documents might have been caused by bad R@M.

Re: 10% of Firefox crashes are caused by bitflips

#203

Earlier quoted context omitted.

That tells you one bit was changed. It doesn't prove that single bit changed due to a hardware failure. It could have been changed by broken software.

Broken software causes null pointer references and similar logic errors. It would be extremely unusual to have an inadvertent ptr ^= (1 that got inserted in the code by accident. That's just not the way that we write software.

Except no one is claiming the bit flip is the pointer vs the data being pointed to or a non pointer value. Given how we write software there’s a lot more bits not in pointer values that still end up “contributing “ to a pointer value. Eg some offset field that’s added to a pointer has a bit flip, the resulting pointer also has a bit flip. But the offset field could have accidentally had a mask applied or a bit set accidentally due to the closeness of & and && or | and ||.

Re: 10% of Firefox crashes are caused by bitflips

#204

People I think are overindexing on this being about "Bad hardware". We have long known that single bit errors in RAM are basically "normal" in terms of modern computers. Google did this research in 2009 to quantify the number of error events in commodity DRAM https://static.googleusercontent.com/media/research.google.c... They found 25,000 to 70,000 errors per billion device hours per Mbit and more than 8% of DIMMs a…

I used to partake in all RAM discussions online. Here, reddit, every technical hardware forum and anywhere workstations were being talked about.

The sentiment was always ECC is a waste and a scam. My goodness the unhinged posts from people who thought it was a trick and couldn't fathom you don't know you're having bits flipped without it. "it's a rip off" without even looking and seeinf often the price was just that of the extra chip.

I've discussed it for 20 years since the first Mac Pro and people just did not want to hear that it had any use. Even after the Google study.

Consumers giving professionals advice. Was same with workstation graphics cards.

Re: 10% of Firefox crashes are caused by bitflips

#205

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

Wow, that’s really interesting! I always suspected bit flips happened undetected way more than we thought, so it’s great to get some real life war stories about it. Also thanks for Guild Wars, many happy hours spent in GW2. :)

Re: 10% of Firefox crashes are caused by bitflips

#206
I bought my PC like 2 weeks ago and ran my ram at 5800 to test its limits and forgot to lower it. After few strange crashes of my fedora desktop - super strange behavior, apps refuse start/stop, can't even escape to the console... I ran memtest today and it lit all red in the first 2 minutes! Then I log in to my stable desktop at 5200 MT and I see this in the front HN page! What are the chances?!!

Re: 10% of Firefox crashes are caused by bitflips

#207

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

I don't understand why ECC memory is not the norm these days. It is only slightly more expensive, but solves all these problems. Some consumer mainboards even support it already.

Re: 10% of Firefox crashes are caused by bitflips

#208
post #134

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

As a community alpha tester of GW1, this was a fun read! Such an educational journey and what a well organized and fruitful one too. We could see the game taking shape before our eyes! As a European, I 100% relied on being young and single with those American time zones. :D Tests could end in my group at like 3 am, lol.

Oh yeah, those were some good times. It was great getting early feedback from you & the other alpha testers, which really changed the course of our efforts.

I remember in the earlier builds we only had a “heal area” spell, which would also heal monsters, and no “resurrect” spell, so it was always a challenge to take down a boss and not accidentally heal it when trying to prevent a player from dying.

Re: 10% of Firefox crashes are caused by bitflips

#209

Going to be downvoted, but I call bullshit on this. Bitflips are frequent (and yes ECC is an improvement but does not solve the problem), but not that frequent. One can either assume users that enabled telemetry are an odd bunch with flaky hardware, or the implementation isnt actually detecting bitflips (potentially, as the messages indicate), but a plathora of problems. Having a 1/10 probability a given struct is ei…

Also having worked in big software with many users, this also doesn't match the data we had.

The only explanation I can see is if Firefox is installed on a user base of incredibly low quality hardware.

Re: 10% of Firefox crashes are caused by bitflips

#210
post #90

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

Are people getting so many FF crashes? Mine rarely does. I leave it running, opening and closing tabs, for weeks on end.

Slack caused frequent FF crashes, until I realized Slack has (had?) a live leak. Added an extension which force-reloads the Slack page every 15 minutes and that stopped the crashing.
Post reply on HN