Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

391–400 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#391

Post by vlovich123 »

rockdoe wrote: post #256
vlovich123 wrote: post #203

Earlier quoted context omitted.

Except no one is claiming the bit flip is the pointer vs the data being pointed to or a non pointer value. Given how we write software there’s a lot more bits not in pointer values that still end up “contributing “ to a pointer value. Eg some offset field that’s added to a pointer has a bit flip, the resulting pointer also has a bit flip. But the offset field could have accidentally had a mask applied or a bit set ac…

I think that if you hit the crash in the same line of code many times, you can safely assume it's your own bug and not a memory issue. If it's only hit once by a random person, memory starts being more likely. (Unless that LOC is scanning memory or smth)

Deduplicating and identifying the source of a crash point is surprisingly hard, to the point that “it’s the only crash of its kind” could be a bug in your logic for linking issues.

Also, in an unsafe language all bets are off. A memory clobber, UAF or race condition can generate quite strange and ephemeral crashes. Even if the majority of time it generates the “same” failure mode, it can still sporadically generate a rare execution trace. It’s best to stop thinking of these as deterministic processes and more as a distribution of possible outcomes.

Re: 10% of Firefox crashes are caused by bitflips

#392

Post by Agingcoder »

RealityVoid wrote: post #381
Agingcoder wrote: post #248

Earlier quoted context omitted.

No it doesn’t :-) I’ve had plenty of servers with faulty ecc dimms that didn’t trigger , and would only show faults when actual memory testing. I had a hard time convincing some of our admins the first time ( ‘no ecc faults you can’t be right ‘ ) but I won the bet. Edit: very old paper by google on these topics. My issues were 6-7 years ago probably. https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

I'm sorry, but I, just like your admins, don't believe this. It's theoretically possible to have "undetectable" errors, but it's very unlikely and you'd see a much higher than this incidence of detected unrecoverable errors and you'd see a much higher incidence than this of repaired errors. I just don't buy the argument of "invisible errors". EDIT: took a look on the paper you linked and it basically says the same th…

Well my admins eventually believed me , so I’m fairly comfortable with what I said.

We also had a few thousands of physical servers with about of terabyte of ram each.

You are right : we did see repaired errors, but we also saw (indirectly, and after testing ) unrepaired ones

Re: 10% of Firefox crashes are caused by bitflips

#393

Post by urxvtcd »

versteegen wrote: post #314
PunchyHamster wrote: post #295

Earlier quoted context omitted.

> Several years later I learned that Dell computers had larger-than-reasonable analog component problems because Dell sourced the absolute cheapest stuff for their computers; I expect that was also a cause. Case in point: I was getting memory errors on my gaming machine, that persisted even after replacing the sticks. It caused windows bluesreen maybe once a month so I kinda lived with it as I couldn't afford to repl…

I'm surprised "faulty PSU" is not on GP's list of common problems. Almost every unstable computer I've ever experienced has been due to either a dying PSU (not an under-specced one) or dying power conversion capacitors on the motherboard.

There's a Polish electronics forum that's infamous because it's kind of actively hostile to them noobs. "Blacklisted power supply, closing thread." is a micro meme at this point.

Re: 10% of Firefox crashes are caused by bitflips

#394

Post by sandworm101 »

likelystory wrote: post #354
versteegen wrote: post #314

Earlier quoted context omitted.

I'm surprised "faulty PSU" is not on GP's list of common problems. Almost every unstable computer I've ever experienced has been due to either a dying PSU (not an under-specced one) or dying power conversion capacitors on the motherboard.

I could see that: - Firefox may be more prevalent on those using Linux, since FF is less “corporate” than Chrome or Edge. - People using Linux are probably putting Linux on old machines that had versions of Windows that are no longer supported. However, what I can’t say next is “PSUs would get old and stop putting out as much” because that doesn’t tend to happen. They just die. Those running Linux on some old tower m…

>> People using Linux are probably putting Linux on old machines

Maybe for linux noobs. But i would suggest that most linux users are not noobs booting a disused pentium from a live CD. They are running linux on the same hardware as windows users. I would further suggest that as anyone installing a not-windows OS is more tech savvy than the average, that linux users actually take better care of thier machines. Linux users take pride in thier machines whereas the average windows user barely knows that computers have fans.

As any linux user for thier specifications and they will quote system reports and memory figues like Marisa Tomei discussing engine timings. Ask a random windows user and they will probably start with the name of the store that sold it.

Re: 10% of Firefox crashes are caused by bitflips

#395

Post by roryirvine »

stnvh wrote: post #359
quesera wrote: post #216

Earlier quoted context omitted.

Describe "demons"? I run four Firefox instances simultaneously, most of the time. No issues to report.

Long hangs / never closes, crash report screen triggers often. macOS. This occurs for me when launching instances from the about:profiles page and using each instance for what I'd describe as normal use

Same, also on macOS. My "personal" firefox profile on my work Macbook Pro, which I use for occasional gmail, HN, wikipedia, and pretty much nothing else, has crashed twice in the last 6 weeks - both times when shutting down to update the OS.

Honestly, I've been blaming MacOS for it since other apps also crashed at the same time (the first time it was Microsoft Intune, the second time it was Slack - I doubt either uses Firefox internally). I don't recall seeing a Firefox crash on my personal laptop running Linux at any point in the past few years.

Re: 10% of Firefox crashes are caused by bitflips

#396

Post by BeetleB »

socalgal2 wrote: post #123
BeetleB wrote: post #90

Earlier quoted context omitted.

Are people getting so many FF crashes? Mine rarely does. I leave it running, opening and closing tabs, for weeks on end.

Does "Weeks on end" = 4? Or do you not take the latest update every 4 weeks?

I run Gentoo, and compile FF from source. I don't think the Gentoo repos update the FF version that frequently. And even if they do and I compile the latest one, I don't automatically quit the existing running version.

Re: 10% of Firefox crashes are caused by bitflips

#397

Post by throwaway85825 »

rafaelmn wrote: post #303
throwaway85825 wrote: post #182

Earlier quoted context omitted.

Very few applications scale with cores. For the vast majority of people single core performance is all they care about, it's also cheaper. They don't need or want workstation gear.

> Very few applications scale with cores You mean like compilers and test suites ? Very few professional workloads don't parallelize well these days.

Only a small percentage of computer users are programmers.

Re: 10% of Firefox crashes are caused by bitflips

#398

Post by gcp »

pflanze wrote: post #379
kleiba wrote: post #246

Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurpris…

Do you happen to use memory resource limits? I used to run Firefox under some, like everything, to prevent it from potentially making the whole system unresponsible, and at the same time had frequent cases of Firefox showing random visual corruptions and crashes. At some point I realized that it was because it was running out of memory, and didn't check malloc failures, thus just continued to run and corrupting memor…

You were seeing issues from the graphics driver, not Firefox.

Any memory allocation failing within the browser forces an instant crash unless the callsite explicitly opts in to handling the allocation failure.

"Check malloc failure" is an opt-out feature in browsers, not opt-in. It's the same in Chromium. Failing to check would cause too many security issues. (One more reason new stuff tends to prefer Rust, etc)

Re: 10% of Firefox crashes are caused by bitflips

#399

Post by sfink »

dboreham wrote: post #81
wmf wrote: post #77

Earlier quoted context omitted.

A common case is a pointer that points to unallocated address space triggers a segfault and when you look at the pointer you can see that it's valid except for one bit.

That tells you one bit was changed. It doesn't prove that single bit changed due to a hardware failure. It could have been changed by broken software.

[I work at Mozilla]

Yes, that's a confounding factor, and in fact the starting assumption when looking at a crash. Sometimes you can be pretty sure it's hardware. For example, if it's a crash on an illegal instruction in non-JITted code, the crash reporter can compare that page of data with the on-disk image that it's supposed to be a read-only copy of. Any mismatches there, especially if they're single bit flips, are much more likely to be hardware.

But I've also seen it several times when the person experiencing the crashes engages on the bug tracker. Often, they'll get weird sporadic but fairly frequent crashes when doing a particular activity, and so they'll initially be absolutely convinced that we have a bug there. But other people aren't reporting the same thing. They'll post a bunch of their crash reports, and when we look at them, they're kind of all over the place (though as they say, almost always while doing some particular thing). Often it'll be something like a crash in the garbage collector while watching a youtube video, and the crashes are mostly the same but scattered in their exact location in the code. That's a good signal to start suspecting bad memory: the GC scans lots of memory and does stuff that is conditional on possibly faulty data. We'll start asking them to run a memory test, at least to rule out hardware problems. When people do it in this situation, it almost always finds a problem. (Many people won't do it, because it's a pain and they're understandably skeptical that we might be sandbagging them and ducking responsibility for a bug. So we don't start proposing it until things start feeling fishy.)

But anyway, that's just anecdata from individual investigations. gsvelto's post is about what he can see at scale.

Re: 10% of Firefox crashes are caused by bitflips

#400

Post by andoando »

bergheim wrote: post #382

Strange. I have a tab hoarding problem, I often have over 1000 tabs open [1][2], and I cannot remember the last time Firefox crashed. I'm thinking it must have been years? I use ublock origin though, which might help since ads do their best to steal your computer and soul through any means possible of course. I also use a bunch of other extensions though, dark reader, vimium, sideberry... I'd expect me to be a bit mo…

I dont get the people with 10+ tabs open drives me crazy, how do you even know whats what?

Just bookmark shit you want to keep!

Post reply on HN