Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

111–120 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#111

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

> Bold claim.

I agree. Good thing he doesn't back up his claim with any sort of evidence or reasoned argument, or you'd look like a huge moron!

Re: 10% of Firefox crashes are caused by bitflips

#112
post #78

ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.

ECC are traditionally slower, quite more complex, and they dont completely eliminate the problem (most memories correct 1 bit per word and detect 2 bits per word). They make sense when environmental factors such as flaky power, temperature or RF interference can be easily discarded - such as a server room. But yeah, I agree with you, as ECC solves like 99% of the cases.

Being able to detect these issues is just as important as preventing them.

Re: 10% of Firefox crashes are caused by bitflips

#113

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

I think they claim that if your computer has bad hardware, you're probably sending a lot of _additional_ crashes to their telemetry system. Your hardware might be working just fine, but the guy next to you might be sending 30% more crashes.

Re: 10% of Firefox crashes are caused by bitflips

#114
post #90

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

Are people getting so many FF crashes? Mine rarely does. I leave it running, opening and closing tabs, for weeks on end.

Its pretty stable for me, except it has some memory leaks. Generally I gotta leave heavy pages open for days at a time to notice, but if I don't close it entirely for over a week or two it will start to chug and crash.

Re: 10% of Firefox crashes are caused by bitflips

#115
post #83
post #78

ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.

It's not even ECC price/availability that bothers me so much, it's that getting CPUs and motherboards that support ECC is non-trivial outside of the server space. The whole consumer class ecosystem is kind of shitty. At least AMD allows consumer class CPUs to kinda sorta use ECC, unlike Intel's approach where only the prosumer/workstation stuff gets ECC.

I've been honestly amazed people actually buy stuff that's not "workstation" gear given IME how much more reliably and consistently it works, but I guess even a generation or two used can be expensive.

Re: 10% of Firefox crashes are caused by bitflips

#116
post #8

is there a way to get the memory tester he mentioned? Is it open source? Once Ram goes bad is there a way or recovering it or is it toasted forever?

This is the best way of marking regions of RAM as bad in Windows:

https://github.com/prsyahmi/BadMemory

I've used it for many years. It only fixes physical hardware faults, not timing errors. For example if a RAM cell is damaged by radiation, not if you're overclocking your RAM.

Re: 10% of Firefox crashes are caused by bitflips

#117
post #88

Earlier quoted context omitted.

Did you/he ever consider redundant allocation for high value content and hash checks for low value assets that are still important? I imagine the largest volume of game memory consumption is media assets which if corrupted would really matter, and the storage requirement for important content would be reasonably negligible?

That's an interesting idea. How might you implement that? Like RAID but on the level of variables? Maybe the one valid use case for getters/setters? :)

As another user fairly pointed out, ECC. But a compiler level flag would probably achieve the redundancy, sourcing stuff from disk etc would probably still need to happen twice to ensure that bit flips do not occur, etc.

Re: 10% of Firefox crashes are caused by bitflips

#118
post #63

I would love to see DDR4 vs DDR5 bitflips. As I understand it DDR5 must come with some level of ECC [1]. [1] https://www.corsair.com/us/en/explorer/diy-builder/memory/is...

From Corsair

>> DDR5 technology comes with an exclusive data-checking feature that serves to improve memory cell reliability and increase memory yield for memory manufacturers. This inclusion doesn't make it full ECC memory though.

"Proper" ECC has a wider memory buss, so the CPU emits checksum bits that are saved alongside every word of memory, and checked again by the CPU when memory is read. Eg. a 64 bit machine would actually have 72 bit memory.

DDR5 "ECC" uses error correction only within the memory stick. It's there to reduce the error rate, so otherwise unacceptable memory is usable - individual cells have become so small that they are not longer acceptably reliable by themselves!

Re: 10% of Firefox crashes are caused by bitflips

#119

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

GW1 was my childhood. The MMO with no monthly fees appealed to my Mom and I met friends for years. The 8 skill build system was genius, as was the cut scenes featuring your player character. If there's ever a 3rd game I would love to see something allowing for more expression through build creation though I could see how that's hard to balance.

Didn't they launch a remake of gw1 recently. Maybe I can get my kids hooked on that instead of this Roblox crap.

Re: 10% of Firefox crashes are caused by bitflips

#120
post #83

Earlier quoted context omitted.

It's not even ECC price/availability that bothers me so much, it's that getting CPUs and motherboards that support ECC is non-trivial outside of the server space. The whole consumer class ecosystem is kind of shitty. At least AMD allows consumer class CPUs to kinda sorta use ECC, unlike Intel's approach where only the prosumer/workstation stuff gets ECC.

I've been honestly amazed people actually buy stuff that's not "workstation" gear given IME how much more reliably and consistently it works, but I guess even a generation or two used can be expensive.

I've had zero issues with AMD's consumer tier of non-WX Threadripper and Ryzen models, FWIW.
Post reply on HN