Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

441–450 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#441

Post by jcalvinowens »

WatchDog wrote: post #138
Animats wrote: post #78

ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.

All DDR5 ram has some amount of error correction built in, because DDR5 is much more prone to bit flipping, it requires it. I'm not really sure if this makes it overall more or less reliable than DDR2/3/4 without ECC though.

DDR5 on-die ECC detects and corrects one-bit errors. It cannot detect two-bit errors, so it will miscorrect some of them into three-bit errors. However, the on-die error correction scheme is specifically specially designed such that the resulting three-bit errors are mathematically guaranteed to be detected as uncorrectable two-bit errors by a standard full system-level ECC running on top of the on-die ECC.

Re: 10% of Firefox crashes are caused by bitflips

#442

Post by quesera »

roryirvine wrote: post #395
stnvh wrote: post #359

Earlier quoted context omitted.

Long hangs / never closes, crash report screen triggers often. macOS. This occurs for me when launching instances from the about:profiles page and using each instance for what I'd describe as normal use

Same, also on macOS. My "personal" firefox profile on my work Macbook Pro, which I use for occasional gmail, HN, wikipedia, and pretty much nothing else, has crashed twice in the last 6 weeks - both times when shutting down to update the OS. Honestly, I've been blaming MacOS for it since other apps also crashed at the same time (the first time it was Microsoft Intune, the second time it was Slack - I doubt either use…

I don't think "crash" is the right word for the Firefox behaviour. Yes it does pop a window that calls itself a "crash reporter", but in my observation it's a shutdown timer timeout that expires after ~60secs.

My guess is that it's trying to obtain or release a filesystem lock, possibly one that it's lost track of in some trivial way.

I've never seen any damage or inconsistencies in the resulting environment. So I don't think it's a dramatic event, just a safety timer that isn't resolved correctly.

Probably a simple, dumb, but harmless bug.

Re: 10% of Firefox crashes are caused by bitflips

#443

Post by jdpage »

alok-g wrote: post #214
ptek wrote: post #164

So does this mean bool true = 3 or should bool true = 5? This will bloat the code a bit.

Interesting. Seems like software could be made a notch more robust by encoding true and false with a larger number of bit differences.

The canonical Boolean values in FORTH are 0 and -1 (that is, all bits set). IIRC the point of that is to unify the bitwise and logical operators, though, not detect bitflips.

Also, at the machine code level, a Boolean controlling a branch or a while loop often doesn't ever make it out of the flags register, where it'll only be a single bit anyway because that's how the hardware works. Not really changeable in software.

Re: 10% of Firefox crashes are caused by bitflips

#444

Post by roryirvine »

quesera wrote: post #442
roryirvine wrote: post #395

Earlier quoted context omitted.

Same, also on macOS. My "personal" firefox profile on my work Macbook Pro, which I use for occasional gmail, HN, wikipedia, and pretty much nothing else, has crashed twice in the last 6 weeks - both times when shutting down to update the OS. Honestly, I've been blaming MacOS for it since other apps also crashed at the same time (the first time it was Microsoft Intune, the second time it was Slack - I doubt either use…

I don't think "crash" is the right word for the Firefox behaviour. Yes it does pop a window that calls itself a "crash reporter", but in my observation it's a shutdown timer timeout that expires after ~60secs. My guess is that it's trying to obtain or release a filesystem lock, possibly one that it's lost track of in some trivial way. I've never seen any damage or inconsistencies in the resulting environment. So I do…

Yes, you're right - the tabs restored fine afterwards and the restart was only delayed for a minute or so, so it was barely even a minor inconvenience.

Contrast that with the dreadful corporate-supplied Edge AI browser I have to use for one client, which seems to randomly close windows without being asked, and never seems to be able to restore them.

Re: 10% of Firefox crashes are caused by bitflips

#445

Post by drob518 »

MrDrMcCoy wrote: post #439
drob518 wrote: post #386

Earlier quoted context omitted.

I concur. A lot of “flakey” issues can be traced to poor quality power supplies. That’s a component that doesn’t get any attention in spec sheets other than a max power rating and I think a lot of manufacturers skimp there. As long as the system boots up and runs for a few minutes, they ship it.

Heck, even dirty power from the wall can contribute. I've seen improvements in stability from putting things behind power conditioners.

Definitely that too, particularly in 2nd-world countries. I remember having a difficult time with dirty power for some hardware products I was responsible for at one time, where the customers were in the Middle East nd Africa in the 1990s. We ended up having to have the PS manufacturer do a redesign to help compensate for dirty power. It can be done, but it costs a bit more.

Re: 10% of Firefox crashes are caused by bitflips

#446

Post by colechristensen »

PunchyHamster wrote: post #298
colechristensen wrote: post #211

Earlier quoted context omitted.

Bit flips do not only happen inside RAM Also, in a game, there is a tremendously large chance that any particular bit flip will have exactly 0 effect on anything. Sure you can detect them, but one pixel being wrong for 1/60th of a second isn't exactly ... concerning. The chance for a bit flip to affect a critical path that is noticeable by the player is very low, and quite a bit lower if you design your game to react…

> The chance for a bit flip to affect a critical path that is noticeable by the player is very low, and quite a bit lower if you design your game to react gracefully. Nobody does > There's a whole practice of writing code for radiation hardened environments that largely consists of strategies for recovering from an impossible to reach state. And again, nobody except stuff that goes to space and few critical machines…

Sure they do, you just have to think about it a different way.

It boils down to exception handling, you don't expect all of your bugs or security vulnerabilities to be known and write your code to be able to react to unplanned states without crashing. Bugs or security vulnerabilities can look a lot like a cosmic ray... a buffer overflow putting garbage in unexpected memory locations vs a cosmic ray putting garbage in unexpected memory locations... a lot of the mitigations are quite the same.

Re: 10% of Firefox crashes are caused by bitflips

#447

Post by seanw444 »

jodrellblank wrote: post #430
netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

This is getting off-topic but I’m amazed by this ability to reach out to computers around the world as a sensor array and infer things we can’t easily find out in other ways. It’s in popular culture and HN comments most often as spyware and mass surveillance of people, and that’s a bit of a shame. GPS location and movement data is what gives Google maps its near-real-time view of traffic on all roads, and busy-ness o…

> It’s in popular culture and HN comments most often as spyware and mass surveillance of people, and that’s a bit of a shame.

I don't know whether you mean it's a shame that people consider it spyware, or if you meant that it's a shame that it manifests as spyware typically. I agree with the latter, not the former. It usually is spyware. If companies went for simple opt-in popups with a brief description of the reasoning, I'd be all for that. I sometimes opt-in to these requests myself, despite being a fairly privacy-conscious person, because I understand the benefit they have to the people collecting the data for good purposes. But when surveillance is opt-out (or no choice given), it's just spyware.

Re: 10% of Firefox crashes are caused by bitflips

#448

Post by pflanze »

gcp wrote: post #398
pflanze wrote: post #379

Earlier quoted context omitted.

Do you happen to use memory resource limits? I used to run Firefox under some, like everything, to prevent it from potentially making the whole system unresponsible, and at the same time had frequent cases of Firefox showing random visual corruptions and crashes. At some point I realized that it was because it was running out of memory, and didn't check malloc failures, thus just continued to run and corrupting memor…

You were seeing issues from the graphics driver, not Firefox. Any memory allocation failing within the browser forces an instant crash unless the callsite explicitly opts in to handling the allocation failure. "Check malloc failure" is an opt-out feature in browsers, not opt-in. It's the same in Chromium. Failing to check would cause too many security issues. (One more reason new stuff tends to prefer Rust, etc)

Thanks for the info! I guess it also makes sense as I realized after posting, if it did use the result of malloc unused it should crash immediately due to references into the zero page segment, thus can't have been what I saw.

Re: 10% of Firefox crashes are caused by bitflips

#449

Post by RealityVoid »

Agingcoder wrote: post #437
RealityVoid wrote: post #431

Earlier quoted context omitted.

Ok, I am sure there is _some_ amount of unrepairable errors. But the initial discussion was that ECC ram makes it go away and your point that it doesn't. And the vast vast majority of the errors, according to my understanding and to the paper you pointed to, are repairable. About 1 out of 400 ish errors are non-repairable. That's a huge improvement! If you had ECC ram, the failures Firefox sees here would drop from 1…

We’re in agreement. I think we diverge on ‘making it go away in my book’. When you’re the one having to debug all these bizarre things ( there were real money numbers involved so these things mattered ), over millions of jobs every day , rare events with low probability don’t disappear - they just happen and take time to diagnose and fix. So in my book ecc improves the situation, but I still had to deal with bad dimm…

Oh, I get this point. If you have a sufficiently large amount of data an you monitor the errors and your software gets better and better even low probability cases will happen and will stand out.

But this is sort of the march of nines.

My knee jerk reaction to blaming ECC is "naaah". Mostly because it's such a convenient scapegoat. It happens, I'm sure, but it would not be the first explanation I reach for. I once heard someone blame "cosmic rays" on a bug that happened multiple times. You can imagine how irked I was on the dang cosmic rays hitting the same data with such consistency!

Anyways, I'm sorry if my tone sounded abrasive, I, too, have appreciated the discussion.

Post reply on HN