Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Flips Your Bit: Cosmic Ray Errors at Mozilla

blog.mozilla.org

Post reply on HN
Print view View original

31–40 of 96 posts

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#31

Post by zepearl »

I don't know folks.

2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it.

Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result => it always matched, I never had any bitflips.

I therefore doubt that the estimation of "1/256MB/month" is correct - I could not prove that, at least not with my laptop.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#32

Post by trollied »

I know HN has a decent Factorio fanbase. Factorio properly stresses PC hardware, and borderline memory is usually ok for a casual gamer until you start a Factorio megabase. A decent example is Warger who does speedruns: https://forums.factorio.com/viewtopic.php?f=7&t=100646 https://www.speedrun.com/factorio#100 Those that have played the game - speedruns are amazing to watch, if you haven't already.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#33

Post by deckard1 »

zepearl wrote: post #31

I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it. Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result…

I've always been a bit skeptical of published numbers. I usually just chalk it up to vastly different operating conditions and scale.

On my home server w/ ECC you can check the corrected and uncorrected (multibit) errors. Assuming my Ryzen is correctly reporting them to Linux, I have 0 errors corrected and 0 uncorrected with a 80 day uptime. I've checked a few other times and never seen an error. Others with ECC often report the same.

My understanding of modern RAM is that it has checks built in to the modules which are somewhat equivalent to ECC already (the correcting part, not the reporting part). Which is a necessity in order to hit the density we are at today.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#34

Post by AshamedCaptain »

robotsteve2 wrote: post #7

Any sort of hardware or software error seems much more likely. Computers are incredibly complex and approximations are used everywhere (in the design of the hardware, in the theory of operation). I don't think inference-based experiments or analysis on cosmic ray bit flips are appropriate. You really need some kind of dedicated cosmic ray detector nearby as a control. If the flux of cosmic rays into the detector is o…

I believe people use "cosmic rays" as catch-all phrase for all these very low probability error causes (just because of the coolness of cosmic rays), but in practice _any_ other cause is much more common than cosmic rays.

Even at the processor level every single transistor on it has a rated mean time between failures a.k.a. MTBF. Sure it may be astronomical, but you do have a lot of transistors, so in practice a random bitflip is not such a rare event. Designers actually explore MTBF vs power usage trade-offs here, and there is even a fascinating area of "fault resilient computing" research.

Every single clock domain crossing has another MTBF (google metastability). Again they are very high (billions of years if done properly), but you will have plenty of such crossings (and the number keeps growing with modern, more asynchronous design).

Processors are quite unreliable things.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#35

Post by tclancy »

zepearl wrote: post #31

I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it. Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result…

>I therefore doubt that the estimation of "1/256MB/month" is correct

As someone who did incredibly poorly in high school physics, this line in the article bothered me as well: the study is from the 1990s when the density of memory would have been much lower. I would think the percentage per megabyte has dropped significantly in 30 or so years. It also assumes a constant form factor for the memory, doesn't it?

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#36

Post by gnufx »

robotsteve2 wrote: post #7

Any sort of hardware or software error seems much more likely. Computers are incredibly complex and approximations are used everywhere (in the design of the hardware, in the theory of operation). I don't think inference-based experiments or analysis on cosmic ray bit flips are appropriate. You really need some kind of dedicated cosmic ray detector nearby as a control. If the flux of cosmic rays into the detector is o…

Yes, but what you'd want to do is look for coincidences between a detector for a cosmic ray shower around (above?) the electronics you're monitoring with whatever it is these days that instruments ECC events. The time resolution would be pathetic for a nuclear physics experiment, but probably good enough.

If you look at the ambient gamma-ray spectrum in a semiconductor detector (which would be germanium rather than silicon) the main background you see is typically from concrete; I'm ashamed to say I've forgotten the energy from K-40, but in the region of 1500 keV. (Ironically, large concrete blocks used for shielding would be regarded as a significant radiation hazard if all the activity in them was concentrated.)

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#37

Post by nixpulvis »

rat9988 wrote: post #22
xenophonf wrote: post #20

Earlier quoted context omitted.

Quoted post unavailable.

> Why not read the linked paper, You answered your own question. The answer is in page 12, which means there is too much information. He is not interested in the whole topic, just about this question. So he asks, maybe someone is charitable enough to answer. Nothing wrong with it.

Skim until you are in the right section?

Literally titled: "Section II - Mitigation of bitsquatting attacks"

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#38

Post by simulate-me »

pitaj wrote: post #19
IncRnd wrote: post #5

"Bitsquatting is a form of cybersquatting which relies on bit-flip errors that occur during the process of making a DNS request. These bit-flips may occur due to factors such as faulty hardware or cosmic rays. When such an error occurs, the user requesting the domain may be directed to a website registered under a domain name similar to a legitimate domain, except with one bit flipped in their respective binary repre…

Is one mitigation TLS certificate verification?

It depends on whether or not the client specifically requested, or is expecting, traffic over HTTPS. But yes, if the user's client requests encrypted traffic, the attacker will not be able to produce a valid certificate. This attack isn't that different than a MITM.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#39

Post by 323 »

bqmjjx0kac wrote: post #21
jldugger wrote: post #12

Earlier quoted context omitted.

Indeed, there was a study in IEEE pointing out the absurdity of cosmic rays as causes -- one point cited was that the vast majority of bitflip happen at specific points in the address space, page boundaries between chips essentially

I'm curious why that is evidence against the cosmic ray explanation. Couldn't it have something to do with the physical layout of memory? Perhaps those page-boundary-adjacent addresses present a larger physical target, perhaps on the bus. Of course I am wildly speculating right now. I'd love to see the article if you have a link!

Modern devices have tiny features which are extremely fragile to any sort of interference, which are much more abundant than cosmic rays.

See the row-hammer attack where you can flip an unrelated bit just by read/writes to adjacent bits from software!!!

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#40

Post by seanw444 »

li2uR3ce wrote: post #10
antognini wrote: post #6

Earlier quoted context omitted.

As they say, there's an app for that: https://cosmicrayapp.com/ Basically it monitors your camera for the streaks produced by cosmic rays. You can see the real time stream of events here: https://cosmicrayobserver.com/#0.4/0/0

Looks like the rays only hit populated areas. https://xkcd.com/1138/

A fascinating revelation indeed.
Post reply on HN