Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

361–370 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#361

Post by Delk »

haspok wrote: post #319
kleiba wrote: post #246

Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurpris…

The most frequent crashes I have with Firefox are when I type in a text area (such as this one right now, or on Reddit, for example). The longer the text I type is, the more probable it is that it's going to crash. Or maybe it doesn't crash, just grinds to such a slow pace that it is equivalent to a crash. My suspicion has always been some kind of a memory leak, but memory corruption also makes sense. Unfortunately,…

I almost never get Firefox crashes on Linux, and I don't remember seeing significant slowdowns with text boxes either, at least not simple ones.

How long are the inputs that you get problems with?

Re: 10% of Firefox crashes are caused by bitflips

#363

Post by aforwardslash »

antonf wrote: post #176
aforwardslash wrote: post #92

Earlier quoted context omitted.

Just updated with a comment. I see firefox crash routinely, so apparently our experiences are quite different :)

Maybe you should check your memory? I recently started to get quite a lot of Firefox crashes, and definitely contributed to this statistic. In the end, the problem was indeed memory - crashes stopped after I tuned down some of the timings. And I used this RAM for a few years with my original settings (XMP profile) without issue.

I experience them in several different devices; On my main device, I have hundreds of chrome tabs and often many workloads running that would be completely corrupt with random bit flips. I'm not discarding the possibility of faulty RAM completely, I just take the measurement of the tweet with a huge grain of salt - after all, I still remember when the FF team constantly denied - for more than half a decade - that the browser had serious memory leak problems, so its not like there isn't a history of pointing out other causes for FF crashes.

Re: 10% of Firefox crashes are caused by bitflips

#364

Post by saagarjha »

grishka wrote: post #287
dpe82 wrote: post #152

Earlier quoted context omitted.

As a mobile dev at YouTube I'd periodically scroll through crash reports associated with code I owned and the long tail/non-clustered stuff usually just made absolutely no sense and I always assumed at least some of it was random bit flips, dodgy hardware, etc.

For the Mastodon Android app, I also sometimes see crashes that make no sense. For example, how about native crashes, on a thread that is created and run by the system, that only contains system libraries in its stack trace, and that never ran any of my code because the app doesn't contain any native libraries to begin with? Unfortunately I've never looked at crashes this way when I worked at VKontakte because there…

Bugs in the system libraries?

Re: 10% of Firefox crashes are caused by bitflips

#365

Post by saagarjha »

funcDropShadow wrote: post #358
m000 wrote: post #316

Earlier quoted context omitted.

Maybe it's high time for some regulation? E.g. EU enforced mandatory USB-C charging from 2025, and pushes for ending production of combustion engine cars by 2035. Why not just make ECC RAM mandatory in new computers starting e.g. from 2030? AMD is already one step away from being compliant. So, it's not an outlandish requirement. And regulating will also force Intel to cut their BS, or risk losing the market.

OMG no. Politician have no business making technological decisions. They make it harder to innovate, i.e. to invent the next generation of ECC with a different name.

Politicians don’t have to be dumb.

Re: 10% of Firefox crashes are caused by bitflips

#366

Post by aforwardslash »

tempaccount5050 wrote: post #157
aforwardslash wrote: post #86

Going to be downvoted, but I call bullshit on this. Bitflips are frequent (and yes ECC is an improvement but does not solve the problem), but not that frequent. One can either assume users that enabled telemetry are an odd bunch with flaky hardware, or the implementation isnt actually detecting bitflips (potentially, as the messages indicate), but a plathora of problems. Having a 1/10 probability a given struct is ei…

How can you possibly be this confident if you don't know the number of times Firefox was run and number of bug reports submitted? Say it's run 100,000,000 times, 1000 reports are submitted, and 10 are bit flips. Seems reasonable. You're misinterpreting what they are saying.

10% of 1000 isnt 10; its 100.And no, its not reasonable - the main reason is that you cannot reliably tell if something is a bit flip or not remotely, because bitflips affect both code and data. Also, 10% of a semi-obscure specific category of failures seems to indicate that the population submitting crashes isn't random enough. I'm a layman in statistics, but this doesn't seem correct, at least not without concrete details on the kinds of bugs being reported and the methodology used. Claiming 10% and being able to demonstrate 10% are different things - and the tweet thread indicates that is this clickbait - something in the lines of "may potentially be a bit-flip". Well, every error may be a bit flip.

Re: 10% of Firefox crashes are caused by bitflips

#367

Post by saagarjha »

shakna wrote: post #145
shevy-java wrote: post #51

> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.

Chromium has better handling for bitflip errors. Mostly due to the Discardable buffers they make such extensive use of. The hardware bugs are there. They're just handled.

By what?

Re: 10% of Firefox crashes are caused by bitflips

#368

Post by sidewndr46 »

netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

Well wow I wasn't expecting to see yet another story from Patrick Wyatt here in the comments! Much appreciated, I've enjoyed reading everything you've written over the years.

Re: 10% of Firefox crashes are caused by bitflips

#369

Post by m000 »

free652 wrote: post #352
m000 wrote: post #316

Earlier quoted context omitted.

Maybe it's high time for some regulation? E.g. EU enforced mandatory USB-C charging from 2025, and pushes for ending production of combustion engine cars by 2035. Why not just make ECC RAM mandatory in new computers starting e.g. from 2030? AMD is already one step away from being compliant. So, it's not an outlandish requirement. And regulating will also force Intel to cut their BS, or risk losing the market.

Cost. You are about to making computers 10-20% more expensive. Computers also aren't used much these days, and phones and tables don't have ECC

ECC has only 10-15% more transistor count. So you're only making one component of the computer 15% more expensive. This should have been a non-brainer, at least before the recent DRAM price hikes.

Also, while computers may not be used much for cosmic rays to be a risk factor, but they're still susceptible to rowhammer-style attacks, which ECC memory makes much harder.

Finally, if you account for the current performance loss due to rowhammer counter-measures, the extra cost of ECC memory is partially offset.

Re: 10% of Firefox crashes are caused by bitflips

#370

Post by crossroadsguy »

tuetuopay wrote: post #285
kleiba wrote: post #246

Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurpris…

Just check your memory with memtest. Two years ago, I've had Factorio crash once on a null pointer exception. I reported the crash to the devs and, likely because the crash place had a null check, they told me my memory was bad. Same as you I said "wait no, no other software ever crashed weirdly on this machine!", but they were adamant. Lo and behold, I indeed had one of my four ram sticks with a few bad addresses. N…

… and are almost always active so that would add to that spread, wouldn’t it?
Post reply on HN