Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

271–280 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#271

Post by tombert »

bpye wrote: post #266
tombert wrote: post #197

Earlier quoted context omitted.

I am not sure I've ever seen a laptop that has ECC memory. I'm sure they exist but I don't think I've seen it. I would definitely like to have a laptop with ECC, because obviously I don't want things to crash and I don't want corrupted data or anything like that, but I don't really use desktop computers anymore.

There are 16" laptops with ECC, you can get a ThinkPad P16 with it for example. I've yet to find any 14" devices with ECC though.

Interesting, I actually have a thinkpad p16s, surprised I didn’t notice ECC availability.

Re: 10% of Firefox crashes are caused by bitflips

#272

Post by iamflimflam1 »

monster_truck wrote: post #178
golem14 wrote: post #122

Earlier quoted context omitted.

Hmm, I wonder if we see, now since we are in a RAM availability crisis, more borderline to bad RAMs creep into the supply chain. If we had a time series graph of this data, it might be revealing.

If you look around you'll see people already putting the new, chinese made DDR4 through its paces, it's holding up far better than anyone expected. Every single time I've had someone pay me to figure out why their build isn't stable, it's always some combination of cheap power supply with no noise filtering, cheap motherboard, and poor cooling. Can't cut corners like that if you want to go fast. That is to say, I've…

The danger is we’ll start to see more QA rejects coming into the market. The temptation to mix in factory rejects into your inventory is going to get very high for a lot of resellers.

Re: 10% of Firefox crashes are caused by bitflips

#273

Post by bpye »

lunar_rover wrote: post #200
_0xdd wrote: post #194

So, why aren't we all using ECC in 2026?

Intel intentionally ripped ECC out of the sweet spot products to charge premium and unfortunately they succeeded. Pentium G4560 supports ECC, Core i7 10700 doesn't.

They did improve this in more recent generations, but you need a W series chipset to use it.

Re: 10% of Firefox crashes are caused by bitflips

#274

Post by c-c-c-c-c »

adonovan wrote: post #198
aforwardslash wrote: post #104

Earlier quoted context omitted.

> In some cases we have even seen crashes in non-memory instructions (e.g. MOV ZR, R1), which implicates misexecution: a fault in the CPU (or a bug in the telemetry bookkeeping, I suppose). Thats the thing. Bit flips impact everything memory-resident - that includes program code. You have no way of telling what instruction was actually read when executing the line your instrumentation may say corresponds to the MOV;…

Good point: I-cache is memory too. (Indeed it is SRAM, so its bits might be even more fragile than DRAM!)

Why would a 6T cell (SRAM) be more fragile than a 1T1C (DRAM) cell?

Re: 10% of Firefox crashes are caused by bitflips

#275

Post by lqet »

kleiba wrote: post #246

Firefox is about the only piece of software in my setup that occasionally crashes. I say "occasionally" for lack of a better word, it's not "all the time", but it is definitely more than I would want to. If that was caused by bad memory, I would expect other software to be similarly affected and hence crash with about comparable frequency. However, it looks like I'm falling more into the other 90% of cases (unsurpris…

> Firefox is about the only piece of software in my setup that occasionally crashes.

I would add Thunderbird to that list.

Re: 10% of Firefox crashes are caused by bitflips

#276

Post by Dylan16807 »

jeffbee wrote: post #132
aforwardslash wrote: post #97

Earlier quoted context omitted.

ECC are traditionally slower, quite more complex, and they dont completely eliminate the problem (most memories correct 1 bit per word and detect 2 bits per word). They make sense when environmental factors such as flaky power, temperature or RF interference can be easily discarded - such as a server room. But yeah, I agree with you, as ECC solves like 99% of the cases.

ECC are "slower" because they are bought by smart people who expect their memory to load the stored value, rather than children who demand racing stripes on the DIMMs.

ECC keeps your bits safe from random flips to a ridiculously large factor. You can run the memory at high consumer speeds, giving up some of that safety margin, while still being more reliable than everything else in your computer.

And there's non-random bit errors that can hit you at any speed, so it's not like going slow guarantees safety.

Re: 10% of Firefox crashes are caused by bitflips

#277

Post by kasabali »

Agingcoder wrote: post #248
Helmut10001 wrote: post #207

Earlier quoted context omitted.

I don't understand why ECC memory is not the norm these days. It is only slightly more expensive, but solves all these problems. Some consumer mainboards even support it already.

No it doesn’t :-) I’ve had plenty of servers with faulty ecc dimms that didn’t trigger , and would only show faults when actual memory testing. I had a hard time convincing some of our admins the first time ( ‘no ecc faults you can’t be right ‘ ) but I won the bet. Edit: very old paper by google on these topics. My issues were 6-7 years ago probably. https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

were they 3-bit flips?

Re: 10% of Firefox crashes are caused by bitflips

#278

Post by Maxion »

conception wrote: post #201
thousand_nights wrote: post #143

Earlier quoted context omitted.

overblown? billions of users use consumer tier hardware just fine. i have servers at home with years of uptime without any ECC memory

But how much bit rot? You’ll never know.

If I don't know about it, then how does it affect me / why should I care? My home server does what it is supposed to do and has done so for a decade. If bit rot /bit flips in memory does not affect my day-to-day life I much prefer cheaper hardware.

I do hope the nuclear powerplant next door uses more fault tolerant hardware, though.

Re: 10% of Firefox crashes are caused by bitflips

#279

Post by aiiane »

netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

I remember one of the first impressions I had in GW1 during test events was the sense of scale in the world that still managed to avoid excessive harsh geometry angles for the most part. Not surprised to hear it was pushing more polygons than average.

P.S. GW1 remains one of my favorite games and the source of many good memories from both PvP and PvE. From fun stories of holding the Hall of Heroes to some unforgettable GvG matches, y'all made a great game.

Re: 10% of Firefox crashes are caused by bitflips

#280

Post by bob1029 »

I've written genetic programming experiments that do not require an explicit mutation operator because the machine would tend to flip bits in the candidate genomes under the heavy system load. It took me a solid week to determine that I didn't actually have a bug in my code. It happens so fast on my machine (when it's properly loaded) that I can depend on it to some extent.
Post reply on HN