Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Flips Your Bit: Cosmic Ray Errors at Mozilla

blog.mozilla.org

Post reply on HN
Print view View original

91–96 of 96 posts

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#91

Post by throwawaylinux »

vlovich123 wrote: post #85
throwawaylinux wrote: post #84

Earlier quoted context omitted.

That's not contrasting evidence. Defects are certainly common sources of error, particularly with cheap commodity components like those used in google's fleet. That does not prove cosmic rays aren't a significant source of SEUs [in any computing device].

No but it adds significant credence against the cosmic ray hypothesis. Particularly since this Mozilla blog post has lots of speculation without any actual data to support the claim. FF is running on even lower cost commodity components without strict controls on the operating environment (eg environmental heat isn’t controlled like it is in data centers, power supplies can be borderline vs carefully species Google m…

It does not add any "credence against" the well established fact that cosmic rays flip bits at significant rates that it has to be designed for.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#92

Post by jldugger »

bqmjjx0kac wrote: post #21
jldugger wrote: post #12

Earlier quoted context omitted.

Indeed, there was a study in IEEE pointing out the absurdity of cosmic rays as causes -- one point cited was that the vast majority of bitflip happen at specific points in the address space, page boundaries between chips essentially

I'm curious why that is evidence against the cosmic ray explanation. Couldn't it have something to do with the physical layout of memory? Perhaps those page-boundary-adjacent addresses present a larger physical target, perhaps on the bus. Of course I am wildly speculating right now. I'd love to see the article if you have a link!

Apologies everyone, the paper I was paper I was thinking of was ACM: "Cosmic Rays Don't Strike Twice" -- https://dl.acm.org/doi/pdf/10.1145/2248487.2150989

The point being that consumer grade memory is straight up error prone, and a bitflip isn't necessarily caused by "cosmic rays" but could just be like, a flaky DRAM chip, network card, etc.

If you are Mozilla it doesn't really matter what the source of bit flips are as much a good understanding of their prevalence and how it might impact your customer experience, telemetry, and even security.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#93

Post by kiririn »

Sirened wrote: post #71
grog454 wrote: post #50

On the subject of bit flips, I am able to detect these in the client to server UDP packets in my game. With specific logging enabled I would see an error about once per minute while receiving about 15,000 of one type of packet per second. I was able to estimate about 1/1,000,000 packets contained a single flipped bit.

I mean, you are using UDP. The packets may have been damaged in transit just due to crumby wiring and not due to cosmic rays corrupting memory

UDP still has checksums. It drops invalid packets

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#94

Post by grog454 »

kiririn wrote: post #93
Sirened wrote: post #71

Earlier quoted context omitted.

I mean, you are using UDP. The packets may have been damaged in transit just due to crumby wiring and not due to cosmic rays corrupting memory

UDP still has checksums. It drops invalid packets

You are right but either the checksum needs to be inspected at application layer or the 1/10^7 was undetected bit flips. At the time I read about general bit flip probability over the internet and it was consistent with what I was seeing. Maybe I can find the paper.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#95

Post by throw10920 »

AshamedCaptain wrote: post #34
robotsteve2 wrote: post #7

Any sort of hardware or software error seems much more likely. Computers are incredibly complex and approximations are used everywhere (in the design of the hardware, in the theory of operation). I don't think inference-based experiments or analysis on cosmic ray bit flips are appropriate. You really need some kind of dedicated cosmic ray detector nearby as a control. If the flux of cosmic rays into the detector is o…

I believe people use "cosmic rays" as catch-all phrase for all these very low probability error causes (just because of the coolness of cosmic rays), but in practice _any_ other cause is much more common than cosmic rays. Even at the processor level every single transistor on it has a rated mean time between failures a.k.a. MTBF. Sure it may be astronomical, but you do have a lot of transistors, so in practice a rand…

Ironically, even though the more modern, "asynchronous" (really, more just asynchronous communication between fully-synchronous clock domains) CPU designs result in more chances for metastability, a fully asynchronous, self-timed design wouldn't have to have any likelihood of metastability at all!

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#96

Post by dextercd »

dextercd wrote: post #51

The '1 error for every 256MB memory a month' sounds like way tko much to me. A program I wrote launches every time I start my computer. It allocates some memory and scans it periodically for unexpected changes. After an equivalent of 15.8 256MB/months no anomalies have been found yet. Would really like to see more authoritative figures for modern consumer hardware.

It did now finally happen, after an equivalent of 18.9 256MB/months.
Post reply on HN