Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

451–460 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#451

Post by stnvh »

quesera wrote: post #442
roryirvine wrote: post #395

Earlier quoted context omitted.

Same, also on macOS. My "personal" firefox profile on my work Macbook Pro, which I use for occasional gmail, HN, wikipedia, and pretty much nothing else, has crashed twice in the last 6 weeks - both times when shutting down to update the OS. Honestly, I've been blaming MacOS for it since other apps also crashed at the same time (the first time it was Microsoft Intune, the second time it was Slack - I doubt either use…

I don't think "crash" is the right word for the Firefox behaviour. Yes it does pop a window that calls itself a "crash reporter", but in my observation it's a shutdown timer timeout that expires after ~60secs. My guess is that it's trying to obtain or release a filesystem lock, possibly one that it's lost track of in some trivial way. I've never seen any damage or inconsistencies in the resulting environment. So I do…

Reassuring to hear I'm not the only one, and would consider this a normal use case for the browser, in fact one of the main reasons I use Firefox over chrome as it's simpler to manage than the latter.

I was hinting in my original comment if these cases are contributing to crash reports in any capacity there is a small chance they could be misattributed towards the claims in the post, especially if memory is not freed correctly on shutdown. Even more so if any memory allocation is shared between processes / helpers.

If I quit normally, don't wait for the "timeout" and force quit I still get the crash report UI immediately which suggests to me something funky going on.

10% is a crazy high percentage to claim for bitflips.

Re: 10% of Firefox crashes are caused by bitflips

#452

Post by Dylan16807 »

rixed wrote: post #405
Dylan16807 wrote: post #268

Earlier quoted context omitted.

If your pieces of important data are very tiny, that's probably your best option. If they're hundreds of bytes or more, then two copies plus two hashes will do a better job.

A single hash should be enough.

Yes, but what's easier depends on layout. "Consensus" makes me think of multiple entire nodes, and in that situation you can have a nice symmetry by making each node store one copy and one small hash.

If you're doing something that's more centralized then one hash might be simpler, but if you're centralized then you should probably use your own error correction codes instead of having multiple copies.

Re: 10% of Firefox crashes are caused by bitflips

#453

Post by [deleted] »

shakna wrote: post #144
nomel wrote: post #94

Earlier quoted context omitted.

I think the most reasonable take would be to just tell the users hardware is borked, they're going to have a bad outside the game too, and point them to one of the many guides around this topic. I don't think engineering effort should ever be put into handling literal bad hardware. But, the user would probably love you for letting them know how to fix all the crashing they have while they use their broken computer! T…

I think I sit in another camp. A lot of my engineering efforts are in working around bad hardware. Better the user sees some lag due to state rebuild versus a crash. Most consumers have what they have, and use what they have. Upgrading everything is now rare. If they got screwed, they'll remain screwed for a few years.

[deleted]

Re: 10% of Firefox crashes are caused by bitflips

#454

Post by yownie »

netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

this exactly the type of stories I come to HN to read, thanks!

Re: 10% of Firefox crashes are caused by bitflips

#455

Post by fastaguy88 »

It is perhaps worth noting that the 25,000 bit flips/out of 470,000 crashes (in a week) are probably not coming from all Firefox users. It would be useful to know how many of those crashes (and bit flips) are happening on the same machine. And whether the crashes/bit flips continue on the same machine continue from week to week.

I can certainly imagine that a very small fraction of Firefox users are generating these results, so that bit flips are not a problem generally.

Re: 10% of Firefox crashes are caused by bitflips

#456

Post by alpaca128 »

alpaca128 wrote: post #99
netcoyote wrote: post #34

Earlier quoted context omitted.

It is huge , but real (see https://news.ycombinator.com/item?id=47258500 ) Browsers, videogames, and Microsoft Excel push computers really hard compared to regular applications, so I expect they're more likely to cause these types of errors. The original Diablo 2 game servers for battle.net, which were Compaq 1U servers, failed at astonishing rates due to their extremely high utilization and consequent heat-generatio…

In my case it doesn't seem to be related to system load. I have an issue where (mainly) using FF can trigger random system freezes on Linux, often with the browser going down first. But running CPU/memory stress tests, compiling things etc don't cause any errors and the cooler is downright bored.

Update: it's starting to look like CPU C-states were the problem.

Re: 10% of Firefox crashes are caused by bitflips

#457

Post by jodrellblank »

seanw444 wrote: post #447
jodrellblank wrote: post #430

Earlier quoted context omitted.

This is getting off-topic but I’m amazed by this ability to reach out to computers around the world as a sensor array and infer things we can’t easily find out in other ways. It’s in popular culture and HN comments most often as spyware and mass surveillance of people, and that’s a bit of a shame. GPS location and movement data is what gives Google maps its near-real-time view of traffic on all roads, and busy-ness o…

> It’s in popular culture and HN comments most often as spyware and mass surveillance of people, and that’s a bit of a shame. I don't know whether you mean it's a shame that people consider it spyware, or if you meant that it's a shame that it manifests as spyware typically. I agree with the latter, not the former. It usually is spyware. If companies went for simple opt-in popups with a brief description of the reaso…

I mean what you did is a shame.

I asked to put the spyware aside for one sub-thread and focus on the astonishing worldwide sensor array, and you talked about the spyware and nothing else.

Re: 10% of Firefox crashes are caused by bitflips

#458

Post by smalley »

thebruce87m wrote: post #282
Agingcoder wrote: post #248

Earlier quoted context omitted.

No it doesn’t :-) I’ve had plenty of servers with faulty ecc dimms that didn’t trigger , and would only show faults when actual memory testing. I had a hard time convincing some of our admins the first time ( ‘no ecc faults you can’t be right ‘ ) but I won the bet. Edit: very old paper by google on these topics. My issues were 6-7 years ago probably. https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

That shouldn’t make sense. It’s not like the ECC info is stored in additional bits separate from the data, it’s built in with the data so you can’t “ignore” it. Hmm, off to read the paper.

The ECC information is stored in separate DRAM devices on the DIMM. This is responsible for some of the increased cost of DIMMs with ECC at a given size. When marketed the extra memory for ECC are typically not included in the size for DIMMs so a 32GB DIMM with and without ECC will have differing numbers of total DRAM devices.

There's a pretty good set of diagrams and descriptions of the faults in this paper https://dl.acm.org/doi/10.1145/3725843.3756089.

Also to the parent: there's an updated public paper on DDR4 era fault observations https://ieeexplore.ieee.org/document/10071066

Re: 10% of Firefox crashes are caused by bitflips

#459

Post by Agingcoder »

RealityVoid wrote: post #449
Agingcoder wrote: post #437

Earlier quoted context omitted.

We’re in agreement. I think we diverge on ‘making it go away in my book’. When you’re the one having to debug all these bizarre things ( there were real money numbers involved so these things mattered ), over millions of jobs every day , rare events with low probability don’t disappear - they just happen and take time to diagnose and fix. So in my book ecc improves the situation, but I still had to deal with bad dimm…

Oh, I get this point. If you have a sufficiently large amount of data an you monitor the errors and your software gets better and better even low probability cases will happen and will stand out. But this is sort of the march of nines. My knee jerk reaction to blaming ECC is "naaah". Mostly because it's such a convenient scapegoat. It happens, I'm sure, but it would not be the first explanation I reach for. I once he…

:-) never forget Occam’s razor !

No you were not abrasive at all - I’ve learned to assume good faith in forum conversations.

In retrospect I should have started by giving the context ( march of 9s is a good description) actually, which would have made everything a lot clearer for everyone.

Re: 10% of Firefox crashes are caused by bitflips

#460

Post by matheusmoreira »

WatchDog wrote: post #138
Animats wrote: post #78

ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.

All DDR5 ram has some amount of error correction built in, because DDR5 is much more prone to bit flipping, it requires it. I'm not really sure if this makes it overall more or less reliable than DDR2/3/4 without ECC though.

ECC also reports error recovery statistics to the operating system. Lets you know if any unrecoverable errors happened. Lets you calculate the error rate which means you can try to predict when your memory modules are going bad.

I think this sort of reporting is a pretty basic feature that should come standard on all hardware. No idea why it's an "enterprise" feature. This market segmentation is extremely annoying and shouldn't exist.

Post reply on HN