Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cosmic rays causing 30k network malfunctions in Japan each year

mainichi.jp

Post reply on HN
Print view View original

41–50 of 120 posts

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#41

Post by viraptor »

kevin_nisbet wrote: post #21
belter wrote: post #10

They have to do two things :-) 1) Use ECC memory 2) Go underground "One experiment measured the soft error rate at the sea level to be 5,950 failures in time (FIT = failures per billion hours) per DRAM chip. When the same test setup was moved to an underground vault, shielded by over 50 feet (15 m) of rock that effectively eliminated all cosmic rays, zero soft errors were recorded.[6] In this test, all other causes o…

> 1) Use ECC memory Not exactly. When I was in telco, where I had this problem was in FPGA's, we had all ECC memory and I never linked any problems to bit flips in RAM. But as I remember, the FPGA's we had were using a type of SRAM cell, but because it's not a memory module the FPGA programming could bit flip. So the product had a checksum function, that read back the program on a cycle and reset itself if the progra…

Compaq/HP handled this with many redundant resources / cores: https://en.wikipedia.org/wiki/Tandem_Computers

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#42

Post by jmcqk6 »

notum wrote: post #16

Cosmic rays are, or rather were, the reason why manufactures avoid shipping high end cameras by plane, cosmic rays kill pixels in imaging sensors: https://youtu.be/98FZ8C6HneE?t=479 It's also the reason why all ISS footage has dead pixels on it.

Does anyone know how this works with our space-based telescopes? Or even our mountain top observatories? Surely if it's this crazy, they have to deal with that as well?

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#43

Post by oconnor663 »

digikata wrote: post #38
shiftpgdn wrote: post #8

Kind of a fun ancedote: In "the old days" when you'd run a server room in your office we had a very large HPC cluster + a significant amount of storage and other one off servers on the top floor of a mid rise office building. I eventually moved it all to a former nuclear fallout facility where our systems were three floors down and under a gigantic pool of water. Error rates and random crashes fell off IMMEDIATELY. I…

A reduction in floor and building vibration might have contributed too?

Or for example, old cables and improperly installed cooling fans might tend to get fixed during a move. Hard to know for sure.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#44

Post by belter »

kevin_nisbet wrote: post #21
belter wrote: post #10

They have to do two things :-) 1) Use ECC memory 2) Go underground "One experiment measured the soft error rate at the sea level to be 5,950 failures in time (FIT = failures per billion hours) per DRAM chip. When the same test setup was moved to an underground vault, shielded by over 50 feet (15 m) of rock that effectively eliminated all cosmic rays, zero soft errors were recorded.[6] In this test, all other causes o…

> 1) Use ECC memory Not exactly. When I was in telco, where I had this problem was in FPGA's, we had all ECC memory and I never linked any problems to bit flips in RAM. But as I remember, the FPGA's we had were using a type of SRAM cell, but because it's not a memory module the FPGA programming could bit flip. So the product had a checksum function, that read back the program on a cycle and reset itself if the progra…

Interesting and makes sense. Do you have any additional references you would suggest and particularly in the context of FPGAs ?

Would you say this quick reference is a good overview ? https://www.intel.com/content/dam/www/programmable/us/en/pdf...

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#45

Post by raisedbyninjas »

belter wrote: post #10

They have to do two things :-) 1) Use ECC memory 2) Go underground "One experiment measured the soft error rate at the sea level to be 5,950 failures in time (FIT = failures per billion hours) per DRAM chip. When the same test setup was moved to an underground vault, shielded by over 50 feet (15 m) of rock that effectively eliminated all cosmic rays, zero soft errors were recorded.[6] In this test, all other causes o…

5950 failures per billion hours is approximately equal to 1 failure every 19 years.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#46

Post by [deleted] »

jmcqk6 wrote: post #42
notum wrote: post #16

Cosmic rays are, or rather were, the reason why manufactures avoid shipping high end cameras by plane, cosmic rays kill pixels in imaging sensors: https://youtu.be/98FZ8C6HneE?t=479 It's also the reason why all ISS footage has dead pixels on it.

Does anyone know how this works with our space-based telescopes? Or even our mountain top observatories? Surely if it's this crazy, they have to deal with that as well?

[deleted]

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#47

Post by btkramer9 »

andy_ppp wrote: post #35
notum wrote: post #28

Earlier quoted context omitted.

Rays hit our planet regardless of Sun, if anything it might block a few. They are caused by ultra-energetic cosmic events, such as star collisions, black hole feeding cycles. Cosmic rays are just very potent photons, capable of knocking out electrons from an atom (meaning they are ionizing) causing havoc in precision electronics and, well, our DNA.

What is the tolerance for cosmic rays for a human... I ask because on the journey to Mars you’ll get quite a few I guess and also on the planet’s surface after you’re there with such a thin atmosphere?

As far as cancer rates go I've heard it's about as bad as taking up smoking cigarettes. Which is not great but doesn't seem like a show stopper

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#48

Post by nightcracker »

raisedbyninjas wrote: post #45
belter wrote: post #10

They have to do two things :-) 1) Use ECC memory 2) Go underground "One experiment measured the soft error rate at the sea level to be 5,950 failures in time (FIT = failures per billion hours) per DRAM chip. When the same test setup was moved to an underground vault, shielded by over 50 feet (15 m) of rock that effectively eliminated all cosmic rays, zero soft errors were recorded.[6] In this test, all other causes o…

5950 failures per billion hours is approximately equal to 1 failure every 19 years.

And if you have a datacenter with 200 machines, that's roughly once a month.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#49

Post by ortusdux »

tyingq wrote: post #33
robbmorganf wrote: post #27

Earlier quoted context omitted.

Have a source? That seems like an awesome way to recycle grocery bags.

The peer comment posted a source that has lots of references. Though most of them are tests in space, where I assume the thickness requirements would rule out grocery bags. I am curious how thick a layer of HDPE you would need on earth to make any notable difference.

That linked nature paper seems to indicate 10g/cm^2 for a 50% reduction. A standard shopping bag is about 5g, so you would need roughly 20k bags/m^2, or ~2k bags/ft^2. At 0.9g/cm^3, that would be roughly 11cm of solid polyethylene

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#50

Post by typon »

tinco wrote: post #39
andy_ppp wrote: post #35

Earlier quoted context omitted.

What is the tolerance for cosmic rays for a human... I ask because on the journey to Mars you’ll get quite a few I guess and also on the planet’s surface after you’re there with such a thin atmosphere?

This is actually a thing people are criticizing Musk for, he is being accused of sending people to Mars with insufficient shielding against cosmic rays. In my opinion it's a bit silly argument, as there's a whole bunch of other risks and quality of life sacrifices made by the persons who are going to undertake that journey. Some raised chance of cancer is probably the least of their worries.

Humans going to Mars should expect to never come back
Post reply on HN