Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Flips Your Bit: Cosmic Ray Errors at Mozilla

blog.mozilla.org

Post reply on HN
Print view View original

41–50 of 96 posts

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#41

Post by nomel »

zepearl wrote: post #31

I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it. Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result…

> I therefore doubt that the estimation of "1/256MB/month" is correct

The probability is related to the physical volume the memory takes, since it's caused by a physical particle going through that volume. So, this rate will continuously drop as memory density increases.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#42

Post by cozzyd »

perihelions wrote: post #9
tconfrey wrote: post #2

I worked at Sonus Networks (now Ribbon[0]) in the early 2000's building VoIP solutions for telcos. We had a bunch of unexplained errors in a new installation in Denver. After much head scratching the engineers on the problem concluded that the higher altitude significantly increased the likelihood of impact by alpha particles and that that was the cause of the problem! (IIRC we increased the shielding on the devices.…

Minor clarification: alpha radiation doesn't come from cosmic rays, rather from U/Th contamination in the circuit materials. The altitude dependent component you'd be seeing is rather muons and neutrons.

you can get alphas from spallation, but you wouldn't really call that alpha radiation.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#44

Post by cozzyd »

zepearl wrote: post #31

I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it. Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result…

A server with 64 GB of ECC ram sitting at an altitude of 3.2 km on the Greenland ice sheet is reporting... 0 bit errors (whether correctable or uncorrectable) in the 244 days it's been up.

A server with 16 GB of ECC ram at an altitude of 3.8 km in California is reporting.... 0 bit errors in the 146 days it's been up.

Maybe I shouldn't believe what /sys/devices/system/edac/mc is reporting? These are EL8 systems...

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#45

Post by cozzyd »

jldugger wrote: post #12
robotsteve2 wrote: post #7

Any sort of hardware or software error seems much more likely. Computers are incredibly complex and approximations are used everywhere (in the design of the hardware, in the theory of operation). I don't think inference-based experiments or analysis on cosmic ray bit flips are appropriate. You really need some kind of dedicated cosmic ray detector nearby as a control. If the flux of cosmic rays into the detector is o…

Indeed, there was a study in IEEE pointing out the absurdity of cosmic rays as causes -- one point cited was that the vast majority of bitflip happen at specific points in the address space, page boundaries between chips essentially

I'd be very interested in reading that article if you have a link (or title, or doi...)

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#46

Post by legalcorrection »

pitaj wrote: post #19
IncRnd wrote: post #5

"Bitsquatting is a form of cybersquatting which relies on bit-flip errors that occur during the process of making a DNS request. These bit-flips may occur due to factors such as faulty hardware or cosmic rays. When such an error occurs, the user requesting the domain may be directed to a website registered under a domain name similar to a legitimate domain, except with one bit flipped in their respective binary repre…

Is one mitigation TLS certificate verification?

Depends on where in the stack the error happened.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#47

Post by ThePhysicist »

One of the first things you'll learn when studying experimental physics is how to come up with all kinds of alternative mechanisms that might explain the result you've observed in your experiment, and then think of ways to test that the results weren't actually caused by those unwanted mechanisms. Most Nobel-prize winning physics experiments were carefully designed to compensate for any relevant secondary effects, and I would even go as far as saying that this is often the largest challenge when doing high-precision experiments.

So the first question I'd ask myself when thinking about cosmic-ray induced errors is how I would ensure that the bit flips are not caused by e.g. problems on the hard drives or the NAND array (which are probably much more likely to occur than cosmic ray events, at least on the surface of the earth).

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#48

Post by mherdeg »

ThePhysicist wrote: post #47

One of the first things you'll learn when studying experimental physics is how to come up with all kinds of alternative mechanisms that might explain the result you've observed in your experiment, and then think of ways to test that the results weren't actually caused by those unwanted mechanisms. Most Nobel-prize winning physics experiments were carefully designed to compensate for any relevant secondary effects, an…

Yeah that was one of the gotchas in the story at https://blogs.oracle.com/linux/post/attack-of-the-cosmic-ray... -- MAYBE it was a bit flip due to a cosmic ray, or MAYBE it was a bit flip due to another layer of the system that makes RAM chips store and retrieve data.

I like the idea of a physicist who thinks about this and says - "well, why should we shrug and say 'maybe it was a cosmic ray?' Surely we can test this! Let's put the computer in a lead-lined enclosure and benchmark the memory failure rate and see if it changes", or whatever.

That's a great extension of the classic computer-hacker view that "of course we can understand why this bug happened, we don't have to shrug and say it segfaults until we restart sometimes, we can just dig some more." How far can you go?

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#49

Post by tedunangst »

simulate-me wrote: post #38
pitaj wrote: post #19

Earlier quoted context omitted.

Is one mitigation TLS certificate verification?

It depends on whether or not the client specifically requested, or is expecting, traffic over HTTPS. But yes, if the user's client requests encrypted traffic, the attacker will not be able to produce a valid certificate. This attack isn't that different than a MITM.

Nothing prevents an attacker from getting a cert for snytimg.com or oslashdot.org.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#50

Post by grog454 »

On the subject of bit flips, I am able to detect these in the client to server UDP packets in my game. With specific logging enabled I would see an error about once per minute while receiving about 15,000 of one type of packet per second. I was able to estimate about 1/1,000,000 packets contained a single flipped bit.
Post reply on HN