Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

371–380 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#371

Post by m000 »

funcDropShadow wrote: post #358
m000 wrote: post #316

Earlier quoted context omitted.

Maybe it's high time for some regulation? E.g. EU enforced mandatory USB-C charging from 2025, and pushes for ending production of combustion engine cars by 2035. Why not just make ECC RAM mandatory in new computers starting e.g. from 2030? AMD is already one step away from being compliant. So, it's not an outlandish requirement. And regulating will also force Intel to cut their BS, or risk losing the market.

OMG no. Politician have no business making technological decisions. They make it harder to innovate, i.e. to invent the next generation of ECC with a different name.

I would argue that in the present conditions, regulation can actually foster and guide real innovation.

With no regulations in place, companies would rather innovate in profit extraction rather improving technology. And if they have enough market capture, they may actually prefer to not innovate, if that would hurt profits.

Re: 10% of Firefox crashes are caused by bitflips

#372

Post by ciupicri »

hikarudo wrote: post #357
Dylan16807 wrote: post #265

Earlier quoted context omitted.

Well for DDR5 that's 25% more chips which isn't great even if you don't get ripped off by market segmentation. It's possible DDR6 will help. If it gets the ability to do ECC over an entire memory access like LPDDR, that could be implemented with as little as 3% extra chip space.

Why 25%, shouldn't it be 12.5%? 8 ECC bits for every 64 bits.

DDR5 ECC RDIMMs (R=registered) have 16 extra bits. From the specifications for Kingston's KSM64R52BS8-16MD [1]:

> x80 ECC (x40, 2 independent I/O sub channels)

On the other hand ECC UDIMMs (U=unbuffered) have only 8. From the specifications for Kingston's KSM56E46BS8KM-16HA [2]:

> x72 ECC (x36, 2 independent I/O sub channels)

Though if I remember correctly, the specifications for the older DDR4 ECC RDIMMs mention only 72 bits.

[1]: https://www.kingston.com/datasheets/KSM64R52BS8-16HA.pdf

[2]: https://www.kingston.com/datasheets/KSM56E46BS8KM-16HA.pdf

Re: 10% of Firefox crashes are caused by bitflips

#373

Post by andrepd »

netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

Amazing story! Reminds me of old gamasutra posts like these https://web.archive.org/web/20170522151205/http://www.gamasu...

Re: 10% of Firefox crashes are caused by bitflips

#374

Post by PaulHoule »

netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

Back in the 90's I had an overclocked AMD486 machine which seemed OK most of the time but had segfaults compiling the Linux kernel. I sent in a bug report and Alan Cox closed it saying it was the fault of my machine being overclocked.

I dialed the machine back to the rated speed but it failed completely within 6 months.

Re: 10% of Firefox crashes are caused by bitflips

#375

Post by thfuran »

kasabali wrote: post #277
Agingcoder wrote: post #248

Earlier quoted context omitted.

No it doesn’t :-) I’ve had plenty of servers with faulty ecc dimms that didn’t trigger , and would only show faults when actual memory testing. I had a hard time convincing some of our admins the first time ( ‘no ecc faults you can’t be right ‘ ) but I won the bet. Edit: very old paper by google on these topics. My issues were 6-7 years ago probably. https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

were they 3-bit flips?

It seems extremely unlikely that you’d end up with a lot of those but no smaller detectable errors.

Re: 10% of Firefox crashes are caused by bitflips

#376

Post by seanalltogether »

lifeisstillgood wrote: post #350

I’m pretty sure Torvalds tells a story of spending days hunting down a compiler bug, only to find it was memory, and then simply never using anything other than EC memory again. 10+% is huge

He specifically mentions this story in the LTT video from a few months ago.

https://youtu.be/mfv0V1SxbNA?si=hS4ZMRYqqLXMkxJW&t=526

Re: 10% of Firefox crashes are caused by bitflips

#377

Post by titzer »

I had a refurbished ThinkPad that had memory corruption. I only noticed because Firefox started to crash an unreasonable amount. Ran memcheck through BIOS and sure enough it was bad RAM.

Have we considered that maybe Firefox is the cause of bad memory?

/s

Re: 10% of Firefox crashes are caused by bitflips

#378

Post by glenstein »

dmos62 wrote: post #348
vultour wrote: post #293

Earlier quoted context omitted.

I spend probably thousands of hours in Firefox every year and I don't think I've ever had it crash.

Same. I don't think I've had a crash in 10+ years.

Same for me, it's simply never crashing for my day to day use. It doesn't mean there aren't idiosyncratic cases out there but anecdata can easily paint any number of pictures.

Re: 10% of Firefox crashes are caused by bitflips

#379

Post by pflanze »

kleiba wrote: post #246

Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurpris…

Do you happen to use memory resource limits? I used to run Firefox under some, like everything, to prevent it from potentially making the whole system unresponsible, and at the same time had frequent cases of Firefox showing random visual corruptions and crashes. At some point I realized that it was because it was running out of memory, and didn't check malloc failures, thus just continued to run and corrupting memory. (That was some 6-8 years ago, maybe Firefox does better now?)

Re: 10% of Firefox crashes are caused by bitflips

#380

Post by BorisMelnik »

versteegen wrote: post #314
PunchyHamster wrote: post #295

Earlier quoted context omitted.

> Several years later I learned that Dell computers had larger-than-reasonable analog component problems because Dell sourced the absolute cheapest stuff for their computers; I expect that was also a cause. Case in point: I was getting memory errors on my gaming machine, that persisted even after replacing the sticks. It caused windows bluesreen maybe once a month so I kinda lived with it as I couldn't afford to repl…

I'm surprised "faulty PSU" is not on GP's list of common problems. Almost every unstable computer I've ever experienced has been due to either a dying PSU (not an under-specced one) or dying power conversion capacitors on the motherboard.

yeah dell consumer pc psus were so awful
Post reply on HN