Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

10% of Firefox crashes are caused by bitflips

mas.to

Post reply on HN
Print view View original

431–440 of 495 posts

Re: 10% of Firefox crashes are caused by bitflips

#431

Post by RealityVoid »

Agingcoder wrote: post #392
RealityVoid wrote: post #381

Earlier quoted context omitted.

I'm sorry, but I, just like your admins, don't believe this. It's theoretically possible to have "undetectable" errors, but it's very unlikely and you'd see a much higher than this incidence of detected unrecoverable errors and you'd see a much higher incidence than this of repaired errors. I just don't buy the argument of "invisible errors". EDIT: took a look on the paper you linked and it basically says the same th…

Well my admins eventually believed me , so I’m fairly comfortable with what I said. We also had a few thousands of physical servers with about of terabyte of ram each. You are right : we did see repaired errors, but we also saw (indirectly, and after testing ) unrepaired ones

Ok, I am sure there is _some_ amount of unrepairable errors.

But the initial discussion was that ECC ram makes it go away and your point that it doesn't. And the vast vast majority of the errors, according to my understanding and to the paper you pointed to, are repairable. About 1 out of 400 ish errors are non-repairable. That's a huge improvement! If you had ECC ram, the failures Firefox sees here would drop from 10% to 0.025%! That is highly significant!

Even more! 2 bit errors now you would be informed of! You would _know_ what is wrong.

You could have 3(!) bit errors and this you might not see, but they'd be several orders of magnitude even rarer.

So yes, it would not 100% go away, but 99.9 % go away. That's... Making it go away in my book.

And last but not least, this paper mentions uncorrectable errors. It says nothing of undetectable ecc errors! You said _undetectable_ errors. I'm sure they happen, but would be surprised if you have any meaningful incidence of this, even at terabytes of data. It's probay on the order of 0.000625 of errors you can get ( but if you want I can do more solid math)

Re: 10% of Firefox crashes are caused by bitflips

#432

Post by dpe82 »

grishka wrote: post #287
dpe82 wrote: post #152

Earlier quoted context omitted.

As a mobile dev at YouTube I'd periodically scroll through crash reports associated with code I owned and the long tail/non-clustered stuff usually just made absolutely no sense and I always assumed at least some of it was random bit flips, dodgy hardware, etc.

For the Mastodon Android app, I also sometimes see crashes that make no sense. For example, how about native crashes, on a thread that is created and run by the system, that only contains system libraries in its stack trace, and that never ran any of my code because the app doesn't contain any native libraries to begin with? Unfortunately I've never looked at crashes this way when I worked at VKontakte because there…

Usually I'd just look at clusters of crashes (those that had similar stack traces) but sometimes when you're running a very small % experiment there's not enough signal so you end up looking at everything. And oh boy was there a lot of noise.

In an app with >billion users you get all kinds of wild stuff.

Re: 10% of Firefox crashes are caused by bitflips

#433

Post by MBCook »

jodrellblank wrote: post #430
netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

This is getting off-topic but I’m amazed by this ability to reach out to computers around the world as a sensor array and infer things we can’t easily find out in other ways. It’s in popular culture and HN comments most often as spyware and mass surveillance of people, and that’s a bit of a shame. GPS location and movement data is what gives Google maps its near-real-time view of traffic on all roads, and busy-ness o…

Doesn’t Google also use the phone accelerometer to try and spot earthquakes?

Re: 10% of Firefox crashes are caused by bitflips

#434

Post by tczMUFlmoNk »

adonovan wrote: post #25

Very interesting. The Go toolchain has an (off by default) telemetry system. For Go 1.23, I added the runtime.SetCrashOutput function and used it to gather field reports containing stack traces for crashes in any running goroutine. Since we enabled it over a year ago in gopls, our LSP server, we have discovered hundreds of bugs. Even with only about 1 in 1000 users enabling telemetry, it has been an invaluable source…

> Even with only about 1 in 1000 users enabling telemetry

How do you know the number/proportion of users who run without telemetry enabled, since by definition you're not collecting their data?

(Not imputing any malice, genuinely curious.)

Re: 10% of Firefox crashes are caused by bitflips

#435

Post by fennecbutt »

netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

That's awesome. But also guild waaars, GW2 I played from beta for years, but it just got boring. Endless expansions with weird story.

We need GW3 already but my fear is mmo as a genre is dying.

Re: 10% of Firefox crashes are caused by bitflips

#436

Post by uncSoft »

fennecbutt wrote: post #435
netcoyote wrote: post #33

I've told this story before on HN, but my biz partner at ArenaNet, Mike O'Brien (creator of battle.net) wrote a system in Guild Wars circa 2004 that detected bitflips as part of our bug triage process, because we'd regularly get bug reports from game clients that made no sense. Every frame (i.e. ~60FPS) Guild Wars would allocate random memory, run math-heavy computations, and compare the results with a table of known…

That's awesome. But also guild waaars, GW2 I played from beta for years, but it just got boring. Endless expansions with weird story. We need GW3 already but my fear is mmo as a genre is dying.

They just need to call it GW Classic apparently and it will sell

Re: 10% of Firefox crashes are caused by bitflips

#437

Post by Agingcoder »

RealityVoid wrote: post #431
Agingcoder wrote: post #392

Earlier quoted context omitted.

Well my admins eventually believed me , so I’m fairly comfortable with what I said. We also had a few thousands of physical servers with about of terabyte of ram each. You are right : we did see repaired errors, but we also saw (indirectly, and after testing ) unrepaired ones

Ok, I am sure there is _some_ amount of unrepairable errors. But the initial discussion was that ECC ram makes it go away and your point that it doesn't. And the vast vast majority of the errors, according to my understanding and to the paper you pointed to, are repairable. About 1 out of 400 ish errors are non-repairable. That's a huge improvement! If you had ECC ram, the failures Firefox sees here would drop from 1…

We’re in agreement.

I think we diverge on ‘making it go away in my book’.

When you’re the one having to debug all these bizarre things ( there were real money numbers involved so these things mattered ), over millions of jobs every day , rare events with low probability don’t disappear - they just happen and take time to diagnose and fix.

So in my book ecc improves the situation, but I still had to deal with bad dimms, and ecc wasn’t enough. We used not to see these issues because we already had too many software bugs, but as we got increasingly reliable, hardware issues slowly became a problem, just like compiler bugs or other elements of the chain usually considered reliable.

I fully agree that there are lots of other cases where this doesn’t matter and ecc is good enough.

Thanks for taking the time to reply !

Re: 10% of Firefox crashes are caused by bitflips

#438

Post by loeg »

zadikian wrote: post #429
loeg wrote: post #418

Earlier quoted context omitted.

And that consumer device should have ECC! That's the whole discussion here.

It's confusing because a few comments up is "for the vast majority of people single core performance is all they care about, it's also cheaper" which is unrelated to ECC.

I think it's coherent -- it's an argument for why most people don't want to buy Workstation class products just to get ECC. (Prices scale with core count. Not linearly, but still.)

Re: 10% of Firefox crashes are caused by bitflips

#439

Post by MrDrMcCoy »

drob518 wrote: post #386
versteegen wrote: post #314

Earlier quoted context omitted.

I'm surprised "faulty PSU" is not on GP's list of common problems. Almost every unstable computer I've ever experienced has been due to either a dying PSU (not an under-specced one) or dying power conversion capacitors on the motherboard.

I concur. A lot of “flakey” issues can be traced to poor quality power supplies. That’s a component that doesn’t get any attention in spec sheets other than a max power rating and I think a lot of manufacturers skimp there. As long as the system boots up and runs for a few minutes, they ship it.

Heck, even dirty power from the wall can contribute. I've seen improvements in stability from putting things behind power conditioners.

Re: 10% of Firefox crashes are caused by bitflips

#440

Post by loeg »

zadikian wrote: post #427
loeg wrote: post #421

Earlier quoted context omitted.

Eventually you might notice the pictures or other documents you were saving on your home server have artifacts, or no longer open. This is undesireable for most people using computer storage. > I much prefer cheaper hardware. The cost savings are modest; order of magnitude 12% for the DIMMs, and less elsewhere. Computers are already extremely cheap commodities.

12% for the DIMMs only, but with Intel you need Xeon and its accompanying motherboard for it. Someone said AMD "kinda" lets you do ECC on consumer hardware, not sure what the caveats are besides just being unbuffered. Assuming that's more due to intentional market segmentation than actual cost, yeah I would pay 12% more for ECC. But I'm with the other guy on not valuing it a ton. I have backups which are needed regar…

> Someone said AMD "kinda" lets you do ECC on consumer hardware, not sure what the caveats are besides just being unbuffered.

That was me. It isn't "officially" supported by AMD, but it should work. You can enable EDAC monitoring in Linux and observe detected correction events happening.

> Assuming that's more due to intentional market segmentation than actual cost

That's the argument, yeah.

Post reply on HN