Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

351–360 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#351

Post by jml7c5 »

WatchDog wrote: post #138
Animats wrote: post #78

ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.

All DDR5 ram has some amount of error correction built in, because DDR5 is much more prone to bit flipping, it requires it. I'm not really sure if this makes it overall more or less reliable than DDR2/3/4 without ECC though.

As I understand it, DDR5's on-die ECC is mostly a cost-saving measure. Rather than fab perfect DRAM that never flips a bit in normal operation (expensive, lower yield), you can fab imperfect DRAM that is expected to sometimes flip, but then use internal ECC to silently correct it. The end result to the user is theoretically the same.

Because you can't track on-die ECC errors, you have no way of knowing how "faulty" a particular DRAM chip is. And if there's an uncorrected error, you can't detect it.

Re: 10% of Firefox crashes are caused by bitflips

#352

Post by free652 »

m000 wrote: post #316
hurfdurf wrote: post #296

Earlier quoted context omitted.

Why? Intel making and keeping it workstation/Xeon-exclusive for a premium for too long. And AMD is still playing along not forcing the issue with their weird "yeah, Zen supports it, but your mainboard may or may not, no idea, don't care, do your own research" stance. These days it's a chicken and egg problem re: price and availability and demand. See also https://news.ycombinator.com/item?id=29838403

Maybe it's high time for some regulation? E.g. EU enforced mandatory USB-C charging from 2025, and pushes for ending production of combustion engine cars by 2035. Why not just make ECC RAM mandatory in new computers starting e.g. from 2030? AMD is already one step away from being compliant. So, it's not an outlandish requirement. And regulating will also force Intel to cut their BS, or risk losing the market.

Cost. You are about to making computers 10-20% more expensive.

Computers also aren't used much these days, and phones and tables don't have ECC

Re: 10% of Firefox crashes are caused by bitflips

#353

Post by d1sxeyes »

Dylan16807 wrote: post #268
d1sxeyes wrote: post #264

Earlier quoted context omitted.

At least three copies, so you can recover based on consensus.

If your pieces of important data are very tiny, that's probably your best option. If they're hundreds of bytes or more, then two copies plus two hashes will do a better job.

Ah, true! You just restore the one that matches its hash. Elegant.

Re: 10% of Firefox crashes are caused by bitflips

#354

Post by likelystory »

versteegen wrote: post #314
PunchyHamster wrote: post #295

Earlier quoted context omitted.

> Several years later I learned that Dell computers had larger-than-reasonable analog component problems because Dell sourced the absolute cheapest stuff for their computers; I expect that was also a cause. Case in point: I was getting memory errors on my gaming machine, that persisted even after replacing the sticks. It caused windows bluesreen maybe once a month so I kinda lived with it as I couldn't afford to repl…

I'm surprised "faulty PSU" is not on GP's list of common problems. Almost every unstable computer I've ever experienced has been due to either a dying PSU (not an under-specced one) or dying power conversion capacitors on the motherboard.

I could see that:

- Firefox may be more prevalent on those using Linux, since FF is less “corporate” than Chrome or Edge.

- People using Linux are probably putting Linux on old machines that had versions of Windows that are no longer supported.

However, what I can’t say next is “PSUs would get old and stop putting out as much” because that doesn’t tend to happen. They just die.

Those running Linux on some old tower may hook up too many devices to an underpowered PSU which could cause problems, but I doubt this is the norm.

If it’s not PSUs, what is it? It’s not electromagnetic radiation doing the bitflipping because that’s too rare.

Maybe bitflips could be caused by low-quality peripherals.

People also don’t vacuum out laptops like they used to vacuum out towers and desktops, so maybe it’s dust.

Or maybe it’s all a ruse and FF is buggy, but they don’t have time to figure it out.

Re: 10% of Firefox crashes are caused by bitflips

#355

Post by epx »

Helmut10001 wrote: post #207
netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

I don't understand why ECC memory is not the norm these days. It is only slightly more expensive, but solves all these problems. Some consumer mainboards even support it already.

And checksummed filesystems.

Re: 10% of Firefox crashes are caused by bitflips

#356

Post by Agingcoder »

thebruce87m wrote: post #282
Agingcoder wrote: post #248

Earlier quoted context omitted.

No it doesn’t :-) I’ve had plenty of servers with faulty ecc dimms that didn’t trigger , and would only show faults when actual memory testing. I had a hard time convincing some of our admins the first time ( ‘no ecc faults you can’t be right ‘ ) but I won the bet. Edit: very old paper by google on these topics. My issues were 6-7 years ago probably. https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

That shouldn’t make sense. It’s not like the ECC info is stored in additional bits separate from the data, it’s built in with the data so you can’t “ignore” it. Hmm, off to read the paper.

I fully agree with you ! Neither soft nor hard memory errors, nothing… but but flips ,and reproducible at that.

We scanned all our machines following this ( a few thousand servers ) and found out that ram issues were actually quite common, as said in the paper.

Re: 10% of Firefox crashes are caused by bitflips

#357

Post by hikarudo »

Dylan16807 wrote: post #265
Helmut10001 wrote: post #207

Earlier quoted context omitted.

I don't understand why ECC memory is not the norm these days. It is only slightly more expensive, but solves all these problems. Some consumer mainboards even support it already.

Well for DDR5 that's 25% more chips which isn't great even if you don't get ripped off by market segmentation. It's possible DDR6 will help. If it gets the ability to do ECC over an entire memory access like LPDDR, that could be implemented with as little as 3% extra chip space.

Why 25%, shouldn't it be 12.5%? 8 ECC bits for every 64 bits.

Re: 10% of Firefox crashes are caused by bitflips

#358

Post by funcDropShadow »

m000 wrote: post #316
hurfdurf wrote: post #296

Earlier quoted context omitted.

Why? Intel making and keeping it workstation/Xeon-exclusive for a premium for too long. And AMD is still playing along not forcing the issue with their weird "yeah, Zen supports it, but your mainboard may or may not, no idea, don't care, do your own research" stance. These days it's a chicken and egg problem re: price and availability and demand. See also https://news.ycombinator.com/item?id=29838403

Maybe it's high time for some regulation? E.g. EU enforced mandatory USB-C charging from 2025, and pushes for ending production of combustion engine cars by 2035. Why not just make ECC RAM mandatory in new computers starting e.g. from 2030? AMD is already one step away from being compliant. So, it's not an outlandish requirement. And regulating will also force Intel to cut their BS, or risk losing the market.

OMG no. Politician have no business making technological decisions. They make it harder to innovate, i.e. to invent the next generation of ECC with a different name.

Re: 10% of Firefox crashes are caused by bitflips

#359

Post by stnvh »

quesera wrote: post #216
stnvh wrote: post #14

Try running two instances of Firefox in parallel with different profiles, then do a normal quit / close operation on one after any use. Demons exist here.

Describe "demons"? I run four Firefox instances simultaneously, most of the time. No issues to report.

Long hangs / never closes, crash report screen triggers often. macOS. This occurs for me when launching instances from the about:profiles page and using each instance for what I'd describe as normal use

Re: 10% of Firefox crashes are caused by bitflips

#360

Post by xxs »

kleiba wrote: post #246

Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurpris…

run y-cruncher if you'd like to test memory and overall stability. It's decent test and a lot better than memtest (in my experience)
Post reply on HN