Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

461–470 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#461

Post by shakna »

saagarjha wrote: post #367
shakna wrote: post #145

Earlier quoted context omitted.

Chromium has better handling for bitflip errors. Mostly due to the Discardable buffers they make such extensive use of. The hardware bugs are there. They're just handled.

By what?

With Discardables. When Blink's allocator detects a fault in a memory section it swaps it out for a new one, and taints the old so it is only reused when no more remains.

Live objects get swapped between Discardable buffers quite frequently. They're not expected to stay at the same position in memory.

Re: 10% of Firefox crashes are caused by bitflips

#462

Post by adonovan »

tczMUFlmoNk wrote: post #434
adonovan wrote: post #25

Very interesting. The Go toolchain has an (off by default) telemetry system. For Go 1.23, I added the runtime.SetCrashOutput function and used it to gather field reports containing stack traces for crashes in any running goroutine. Since we enabled it over a year ago in gopls, our LSP server, we have discovered hundreds of bugs. Even with only about 1 in 1000 users enabling telemetry, it has been an invaluable source…

> Even with only about 1 in 1000 users enabling telemetry How do you know the number/proportion of users who run without telemetry enabled, since by definition you're not collecting their data? (Not imputing any malice, genuinely curious.)

Good question. We don't know the true figure, but we extrapolate the denominator from estimates of the total number of Go users and the fraction of Go users that run gopls.

Re: 10% of Firefox crashes are caused by bitflips

#463

Post by thebruce87m »

smalley wrote: post #458
thebruce87m wrote: post #282

Earlier quoted context omitted.

That shouldn’t make sense. It’s not like the ECC info is stored in additional bits separate from the data, it’s built in with the data so you can’t “ignore” it. Hmm, off to read the paper.

The ECC information is stored in separate DRAM devices on the DIMM. This is responsible for some of the increased cost of DIMMs with ECC at a given size. When marketed the extra memory for ECC are typically not included in the size for DIMMs so a 32GB DIMM with and without ECC will have differing numbers of total DRAM devices. There's a pretty good set of diagrams and descriptions of the faults in this paper https://…

I think you responded to the wrong person, unless you think I was implying that the extra bits needed for ECC didn’t need extra space at all? I wasn’t suggesting that - just that they aren’t like a checksum that is stored elsewhere or something that can be ignored - the whole 72 bits are needed to decode the 64 bits of data and the 64 bits of data cannot be read independently.

Re: 10% of Firefox crashes are caused by bitflips

#464

Post by smalley »

thebruce87m wrote: post #463
smalley wrote: post #458

Earlier quoted context omitted.

The ECC information is stored in separate DRAM devices on the DIMM. This is responsible for some of the increased cost of DIMMs with ECC at a given size. When marketed the extra memory for ECC are typically not included in the size for DIMMs so a 32GB DIMM with and without ECC will have differing numbers of total DRAM devices. There's a pretty good set of diagrams and descriptions of the faults in this paper https://…

I think you responded to the wrong person, unless you think I was implying that the extra bits needed for ECC didn’t need extra space at all? I wasn’t suggesting that - just that they aren’t like a checksum that is stored elsewhere or something that can be ignored - the whole 72 bits are needed to decode the 64 bits of data and the 64 bits of data cannot be read independently.

If we're talking about standard server RDIMMs with ECC (or the prosumer stuff) the CPU visible ECC (excluding DDR5's on-die ECC) is typically implemented as a sideband value you could ignore if you disabled the correction logic.

I suppose what winds up where is up to the memory controller but (for DDR5) in each BL16 transaction beat you're usually getting 32 bits of data value and 8 bits of ECC (per sub channel). Those ECC bits are usually called check bits CB[7:0] and they accompany the data bits DQ[31:0] .

If you're talking about transactions for LPDDR things are a bit different there, though as that has to be transmitted inband with your data

Re: 10% of Firefox crashes are caused by bitflips

#465

Post by sandworm101 »

PaulDavisThe1st wrote: post #426
sandworm101 wrote: post #394

Earlier quoted context omitted.

>> People using Linux are probably putting Linux on old machines Maybe for linux noobs. But i would suggest that most linux users are not noobs booting a disused pentium from a live CD. They are running linux on the same hardware as windows users. I would further suggest that as anyone installing a not-windows OS is more tech savvy than the average, that linux users actually take better care of thier machines. Linux…

Unix user for 35 years, Linux for 30+ years ... my case fan died during the summer of last year ... just took the side panel off and kept things running. So much for taking pride in my machine :)

An exception to prove the rule. You fixed it yourself and are here proud of your machine.

I did basically the same thing recently when I built an AI rig. I tried to put it in a sever rack case but the fan noise was too much. So I ditched the rack and put in an open mining frame.

Re: 10% of Firefox crashes are caused by bitflips

#466

Post by stubish »

mock-possum wrote: post #422
BorisMelnik wrote: post #380

Earlier quoted context omitted.

yeah dell consumer pc psus were so awful

Which is kinda crazy to me, in light of how durable their business laptops have been in my experience. I’ve owned maybe 6 pc laptops in my career, and the only 2 that’ve survived that nearly 20 year space are both dells.

Does Dell design and/or build their own laptops? Depending on the year it is likely just their brand and specs, designed and built by an ODM.

Re: 10% of Firefox crashes are caused by bitflips

#467

Post by thebruce87m »

smalley wrote: post #464
thebruce87m wrote: post #463

Earlier quoted context omitted.

I think you responded to the wrong person, unless you think I was implying that the extra bits needed for ECC didn’t need extra space at all? I wasn’t suggesting that - just that they aren’t like a checksum that is stored elsewhere or something that can be ignored - the whole 72 bits are needed to decode the 64 bits of data and the 64 bits of data cannot be read independently.

If we're talking about standard server RDIMMs with ECC (or the prosumer stuff) the CPU visible ECC (excluding DDR5's on-die ECC) is typically implemented as a sideband value you could ignore if you disabled the correction logic. I suppose what winds up where is up to the memory controller but (for DDR5) in each BL16 transaction beat you're usually getting 32 bits of data value and 8 bits of ECC (per sub channel). Tho…

We are talking about errors happening in user space applications with ECC operating normally and what the application ultimately sees.

My point is that when writing an app you wouldn’t be able to “not use” ECC accidentally or easily if it’s there. It’s just seamless. I’m not talking about special test modes or accessing stuff differently on purpose.

Interesting that DDR5 is different than DDR4. 8 bits for 32 is doubling of 8 for 64 so it must have been warranted.

Re: 10% of Firefox crashes are caused by bitflips

#468

Post by rafaelmn »

loeg wrote: post #418
rafaelmn wrote: post #385

Earlier quoted context omitted.

But those tools aren't really compute bound anyway - you're not buying a workstation to do them, you're getting a consumer laptop or a tablet.

And that consumer device should have ECC! That's the whole discussion here.

Why ? If your device is a thin client for web services/gaming the risk of bitflips/bad ram is a minor annoyance.

Re: 10% of Firefox crashes are caused by bitflips

#469

Post by rincebrain »

kevincox wrote: post #412
rincebrain wrote: post #41

Earlier quoted context omitted.

The example I gave in there is of adding sentinel values in your data, so you can check the constants in your data structures later and go "oh, this is overwritten with garbage" versus "oh, this is one or two bits off". I would imagine plumbing things like that through most common structures is what was done there, though I haven't done the archaeology to find out, because Firefox is an enormous codebase to try and f…

This doesn't always protect against out-of-bounds writes. Although if these sentinel values are in read only memory mappings it probably gets pretty close. (Especially if you consider kernel memory corruption a "bitflip".)

Well, yes, the author described it as "a heuristic", which checking sentinel values would seem to match.

Re: 10% of Firefox crashes are caused by bitflips

#470

Post by rincebrain »

patrulek wrote: post #233
rincebrain wrote: post #41

Earlier quoted context omitted.

The example I gave in there is of adding sentinel values in your data, so you can check the constants in your data structures later and go "oh, this is overwritten with garbage" versus "oh, this is one or two bits off". I would imagine plumbing things like that through most common structures is what was done there, though I haven't done the archaeology to find out, because Firefox is an enormous codebase to try and f…

But it would be also possible that sentinel value used for comparison changed because of bitflip, not data structure used by program.

Detecting that is the point, yes.
Post reply on HN