[flagged]
Operating systems use less RAM than Firefox.
10% of Firefox crashes are caused by bitflips
161–170 of 495 posts
Re: 10% of Firefox crashes are caused by bitflips
#162> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.
I haven't seen a single firefox or chrome crash in months now, you should really stress-test your hardware.
firefox crashes... decently often for me, but it's usually pretty clear what the cause is [having a bunch of other programs open]. every time i can recall my computer bluescreening [in the last year~, since that's how long ive had it] it was because of firefox tho.
this may have something to do with the fact that my laptop is from 2017, however.
Re: 10% of Firefox crashes are caused by bitflips
#163I would love to see DDR4 vs DDR5 bitflips. As I understand it DDR5 must come with some level of ECC [1]. [1] https://www.corsair.com/us/en/explorer/diy-builder/memory/is...
DDR5 comes with marginal DRAM that is patched up with ECC to boost yields. It's not the same as fully reliable RAM.
The net error rate is lower with the internal ECC.
DDR4 is not fully reliable memory either.
This is common for many high speed electrical engineering challenges: Running a slightly higher error rate option with ECC on top can have an overall lower error rate at higher throughput than the alternative of running it slow enough to push the error rate down below some threshold.
It makes some people nervous because they don’t like the idea of errors being corrected, but the system designers are looking at overall error rates. The ECC is included in the system’s operation so it isn’t something that is worthwhile to separate out.
Re: 10% of Firefox crashes are caused by bitflips
#164So does this mean bool true = 3 or should bool true = 5?
This will bloat the code a bit.
Re: 10% of Firefox crashes are caused by bitflips
#165Earlier quoted context omitted.
GW1 was my childhood. The MMO with no monthly fees appealed to my Mom and I met friends for years. The 8 skill build system was genius, as was the cut scenes featuring your player character. If there's ever a 3rd game I would love to see something allowing for more expression through build creation though I could see how that's hard to balance.
Didn't they launch a remake of gw1 recently. Maybe I can get my kids hooked on that instead of this Roblox crap.
For what it's worth, Roblox is how I discovered code at age 10.
Re: 10% of Firefox crashes are caused by bitflips
#166I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…
Every interesting bug report I've read about Guild Wars is Dwarf Fortress tier. A very hardcore, longtime player who was recounting some of the better ones to me shared a most excellent one wrt spirits or ghosts, some sort of player summoned thing that were sticking around endlessly and causing OOM errors?
Re: 10% of Firefox crashes are caused by bitflips
#167> In other words up to 10% of all the crashes Firefox users see are not software bugs, they're caused by hardware defects! Bold claim. From my gut feeling this must be incorrect; I don't seem to get the same amount of crashes using chromium-based browsers such as thorium.
What do you mean "the same amount"? If your browser never crashes, 10% of zero is zero.
Re: 10% of Firefox crashes are caused by bitflips
#168ECC should have become standard around the time memories passed 1GB. It's seriously annoying that ECC memory is hard to get and expensive, but memory with useless LEDs attached is cheap.
ECC is standard at this point (current RAM flips so many bits it's basically mandatory). Also, most CPUs have "machine checks" that are supposed to detect incorrect computations + alert the OS.
However, there are still gaps. For one thing, the OS has to be configured to listen for + act on machine check exceptions.
On the hardware level, there's an optional spec to checksum the link between the CPU and the memory. Since it's optional, many consumer machines do not implement it, so then they flip bits not in RAM, but on the lines between the RAM and the CPU.
It's frustrating that they didn't mandate error detection / correction there, but I guess the industry runs on price discrimination, so most people can't have nice things.
Re: 10% of Firefox crashes are caused by bitflips
#169A 5 part thread where they say they're "now 100% positive" the crashes are from bitflips, yet not a single word is spent on how they're supposedly detecting bitflips other than just "we analyze memory"?
The simplest way to do this, what I believe memtest86 and friends do, is to write a fixed pattern over a region of memory and then read it back later and see if it changed; then you write patterns that require flipping the bits that you wrote before, and so on. Things like [1] will also tell you that something corrupted your memory, and if you see a nontrivial (e.g. lots of bits high and low) magic number that has on…
[4] looks like it's only a runner for the actual testing, which is a separate crate: https://github.com/mozilla/memtest
(see: https://github.com/mozilla-firefox/firefox/blob/main/toolkit..., which points to a specific commit in that repo - turns out to be tip of main)
Re: 10% of Firefox crashes are caused by bitflips
#170Very interesting. The Go toolchain has an (off by default) telemetry system. For Go 1.23, I added the runtime.SetCrashOutput function and used it to gather field reports containing stack traces for crashes in any running goroutine. Since we enabled it over a year ago in gopls, our LSP server, we have discovered hundreds of bugs. Even with only about 1 in 1000 users enabling telemetry, it has been an invaluable source…
You might consider adding the CPU temperature to the report, if there's a reasonable way to get it (haven't tried inside a VM). Then you could at least filter out extremely hot hardware.
CPU model / stepping / microcode versions are probably at least as useful as temperature. I'd also try to get things like the actual DRAM timing + voltage vs. what the XMP extensions (or similar) advertise the manufacturer tested the memory at.
I have at least one motherboard that just re-auto-overclocks itself into a flaky configuration if boot fails a few times in a row (which can happen due to loose power cords, or whatever).