Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Flips Your Bit: Cosmic Ray Errors at Mozilla

blog.mozilla.org

Post reply on HN
Print view View original

61–70 of 96 posts

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#61

Post by Beltalowda »

323 wrote: post #39
bqmjjx0kac wrote: post #21

Earlier quoted context omitted.

I'm curious why that is evidence against the cosmic ray explanation. Couldn't it have something to do with the physical layout of memory? Perhaps those page-boundary-adjacent addresses present a larger physical target, perhaps on the bus. Of course I am wildly speculating right now. I'd love to see the article if you have a link!

Modern devices have tiny features which are extremely fragile to any sort of interference, which are much more abundant than cosmic rays. See the row-hammer attack where you can flip an unrelated bit just by read/writes to adjacent bits from software!!!

This is also why memtest has all these testing patterns, and one reason why you typically want to leave it running overnight (or as long as feasible) instead of "yup, we read and wrote all bits and it's all fine!"

Row hammer is one of those things that took advantage of something everyone already kind of knew, just never thought about as a security problem.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#62

Post by xenophonf »

rat9988 wrote: post #29
xenophonf wrote: post #27

Earlier quoted context omitted.

How could I possibly answer their question better than the experts who wrote the paper?

Maybe you can't but someone else can. The question is open to anyone who can and wants to answer.

But the answer is right there, section 2, starting on page 12 of the linked paper.

We're hackers. Autodidacticism is our creed. Research papers are our bread and butter. Why settle for speculation from the pseudonymous masses when a rich vein of well written analysis by reputable authors is one click away?

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#63

Post by exikyut »

joatmon-snoo wrote: post #56
zepearl wrote: post #31

I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it. Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result…

The last time I did research into this, this was the most concrete real world example I could find, which works out to ~41 bitflips/GiB-month: > Jaguar had 360 terabytes of main memory [and] was logging ECC errors at a rate of 350 per minute Source: https://spectrum.ieee.org/how-to-kill-a-supercomputer-dirty-...

This is throwing up a soft paywall for me FWIW - I get up to "...couldn’t run more than an hour or so without crashing." and then the article text stops, there's a "become a member" interstitial and then I'm fed with tons of headlines for different articles instead.

Because of the sharp contrast with the interstitial block I thought it was like a footer, and then a skippable/ignorable part of the layout, and spent some time confusedly scrolling up and down thinking the content was past it, then I realized the "Keep Reading" bits were links to other articles... wow what a mess.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#64

Post by ChrisLomont »

tclancy wrote: post #35
zepearl wrote: post #31

I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it. Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result…

>I therefore doubt that the estimation of "1/256MB/month" is correct As someone who did incredibly poorly in high school physics, this line in the article bothered me as well: the study is from the 1990s when the density of memory would have been much lower. I would think the percentage per megabyte has dropped significantly in 30 or so years. It also assumes a constant form factor for the memory, doesn't it?

The energy needed to flip a bit has drastically lowered since the 1990s, and increased density makes bit flip rates increase at some point since a single physical event can flip multiple bits.

So it's not so simple.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#66

Post by Rebelgecko »

nomel wrote: post #41
zepearl wrote: post #31

I don't know folks. 2 years ago I took a laptop which I wasn't using (16 GiB RAM non-ECC) => I created in Linux with Python an array ("bytes"? Don't remember exactly anymore) of ~10 or 12 GiB containing random integers => computed the array's hash and saved it. Then for ~1-2 months I recomputed from time to time the hash of that array (inbetween the laptop was in suspend-to-RAM) and compared it to the original result…

> I therefore doubt that the estimation of "1/256MB/month" is correct The probability is related to the physical volume the memory takes, since it's caused by a physical particle going through that volume. So, this rate will continuously drop as memory density increases.

My understanding is that newer/denser transistors are actually more susceptible to single event upsets, because it takes less energy to flip a bit.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#67

Post by twoodfin »

dextercd wrote: post #51

The '1 error for every 256MB memory a month' sounds like way tko much to me. A program I wrote launches every time I start my computer. It allocates some memory and scans it periodically for unexpected changes. After an equivalent of 15.8 256MB/months no anomalies have been found yet. Would really like to see more authoritative figures for modern consumer hardware.

Google did a pretty comprehensive study on their data center DRAM a decade ago:

https://static.googleusercontent.com/media/research.google.c...

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#68

Post by tclancy »

ChrisLomont wrote: post #64
tclancy wrote: post #35

Earlier quoted context omitted.

>I therefore doubt that the estimation of "1/256MB/month" is correct As someone who did incredibly poorly in high school physics, this line in the article bothered me as well: the study is from the 1990s when the density of memory would have been much lower. I would think the percentage per megabyte has dropped significantly in 30 or so years. It also assumes a constant form factor for the memory, doesn't it?

The energy needed to flip a bit has drastically lowered since the 1990s, and increased density makes bit flip rates increase at some point since a single physical event can flip multiple bits. So it's not so simple.

Thanks. So I was sort of right only in the exact opposite direction!

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#69

Post by icapybara »

bqmjjx0kac wrote: post #21
jldugger wrote: post #12

Earlier quoted context omitted.

Indeed, there was a study in IEEE pointing out the absurdity of cosmic rays as causes -- one point cited was that the vast majority of bitflip happen at specific points in the address space, page boundaries between chips essentially

I'm curious why that is evidence against the cosmic ray explanation. Couldn't it have something to do with the physical layout of memory? Perhaps those page-boundary-adjacent addresses present a larger physical target, perhaps on the bus. Of course I am wildly speculating right now. I'd love to see the article if you have a link!

This is tangential, but "Cosmic rays do not cause bit flips at a significant rate" is the default position and does not need supporting evidence.

It's like if I claimed that planes occasionally crash due to a local density of dark matter pulling them down. Nobody would need to provide evidence against my theory, it's outrageous. I'm the one who has to provide the evidence.

Re: What Flips Your Bit: Cosmic Ray Errors at Mozilla

#70

Post by octoberfranklin »

aaaaaaaaaaab wrote: post #59
dahfizz wrote: post #54

Earlier quoted context omitted.

How does a comic bit flip make it past the Ethernet CRC?

It occurs in your machine before hitting the network.

because your machine vendor was too cheap to use ECC DRAM.
Post reply on HN