Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

131–140 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#132
post #78

ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.

ECC are traditionally slower, quite more complex, and they dont completely eliminate the problem (most memories correct 1 bit per word and detect 2 bits per word). They make sense when environmental factors such as flaky power, temperature or RF interference can be easily discarded - such as a server room. But yeah, I agree with you, as ECC solves like 99% of the cases.

ECC are "slower" because they are bought by smart people who expect their memory to load the stored value, rather than children who demand racing stripes on the DIMMs.

Re: 10% of Firefox crashes are caused by bitflips

#133
post #63

I would love to see DDR4 vs DDR5 bitflips. As I understand it DDR5 must come with some level of ECC [1]. [1] https://www.corsair.com/us/en/explorer/diy-builder/memory/is...

DDR5 comes with marginal DRAM that is patched up with ECC to boost yields. It's not the same as fully reliable RAM.

Similar to CPUs, where many arrays have spare yield capacity, even whole cores can get disabled (and possibly sold in a different bin). DRAM stores redundant electrons in capacitors to patch it up and boost yields. Everything in reliability is a spectrum.

"ECC" does not give you fully reliable RAM. UEs are still be observed.

What's the chance of fail? If you have one device that achieves equal performance with less reliable cells and redundancy to another device that uses more reliable cells without redundancy, it's not really any different.

NAND is horribly flaky, cell errors are a matter of course. You could buy boutique NOR or SLC NAND or something if you want really good cells. You wouldn't though, because it would be ruinously expensive, but also it would not really give you a result that an SSD with ECC can't achieve.

Re: 10% of Firefox crashes are caused by bitflips

#134

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

As a community alpha tester of GW1, this was a fun read! Such an educational journey and what a well organized and fruitful one too. We could see the game taking shape before our eyes! As a European, I 100% relied on being young and single with those American time zones. :D Tests could end in my group at like 3 am, lol.

Re: 10% of Firefox crashes are caused by bitflips

#135

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

I haven't seen a single firefox or chrome crash in months now, you should really stress-test your hardware.

Re: 10% of Firefox crashes are caused by bitflips

#136

Earlier quoted context omitted.

GW1 was my childhood. The MMO with no monthly fees appealed to my Mom and I met friends for years. The 8 skill build system was genius, as was the cut scenes featuring your player character. If there's ever a 3rd game I would love to see something allowing for more expression through build creation though I could see how that's hard to balance.

Didn't they launch a remake of gw1 recently. Maybe I can get my kids hooked on that instead of this Roblox crap.

Yes, they did relaunch it as Guild Wars Reforged with Steam Deck and controller support and other changes

https://wiki.guildwars.com/wiki/Guild_Wars_Reforged

Re: 10% of Firefox crashes are caused by bitflips

#137
post #87

Earlier quoted context omitted.

It says 10% of crashes If Firefox itself has so few bugs that it crashes very infrequently, it is not contradictory to what you are saying. I wouldn't be surprised if 99% of crashes in my "hello world" script are caused by bit flips.

Just updated with a comment. I see firefox crash routinely, so apparently our experiences are quite different :)

You should look at about:crashes and see if there's any commonality in the causes, or bugs associated with them (though often bugs won't be associated with the crash if it isn't filed from crash-stats or have the crash signature in the bug)

Re: 10% of Firefox crashes are caused by bitflips

#138
post #78

ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.

All DDR5 ram has some amount of error correction built in, because DDR5 is much more prone to bit flipping, it requires it.

I'm not really sure if this makes it overall more or less reliable than DDR2/3/4 without ECC though.

Re: 10% of Firefox crashes are caused by bitflips

#139

>That fancy ARM-based MacBook with RAM soldered on the CPU package? We've got plenty of crashes from those, good luck replacing that RAM without super-specialized equipment and an extraordinarily talented technician doing the job. CPU caches and registers - how exactly are they different from a RAM on a SoC in this regard?

In just about every way. CPU caches are made from SRAM and live on the CPU itself. Main system RAM is made from DRAM and live on separate chips even if they are soldered into the same physical package (system in package or SiP). The RAM still isn't on the SoC.

Unless its gpu

Re: 10% of Firefox crashes are caused by bitflips

#140
post #72

Definitely going to hard disagree with Gabriele Svelto's take. I could point to the comments, however, let me bring up my own experiences across personal devices and organizational devices. In particular, note where he says this: "I can't answer that question directly because crash reports have been designed so that they can't be tracked down to a single user. I could crunch the data to find the ones that are likely…

force-kills like sudo reboot will show UI on restart indicating it didn't shut down cleanly, but that isn't reported as a crash. You can see how often you actually crash via about:crashes (and also see what happened)
Post reply on HN