Live data from Hacker News

Cosmic rays causing 30k network malfunctions in Japan each year

mainichi.jp

21–30 of 120 posts

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#21
post #10

They have to do two things :-) 1) Use ECC memory 2) Go underground "One experiment measured the soft error rate at the sea level to be 5,950 failures in time (FIT = failures per billion hours) per DRAM chip. When the same test setup was moved to an underground vault, shielded by over 50 feet (15 m) of rock that effectively eliminated all cosmic rays, zero soft errors were recorded.[6] In this test, all other causes o…

> 1) Use ECC memory

Not exactly. When I was in telco, where I had this problem was in FPGA's, we had all ECC memory and I never linked any problems to bit flips in RAM. But as I remember, the FPGA's we had were using a type of SRAM cell, but because it's not a memory module the FPGA programming could bit flip. So the product had a checksum function, that read back the program on a cycle and reset itself if the program no longer matched the checksum. So we would see 1-2 crashes / restarts per week in our FPGAs that we believe were bit flips.

We then ran an anlysis on any of these that higher than expected error rates to try and identify actually bad hardware and replace them.

I think the vendor eventually came up with a way to reprogram the FPGA without just crashing and rebooting the entire board.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#22

Kind of a fun ancedote: In "the old days" when you'd run a server room in your office we had a very large HPC cluster + a significant amount of storage and other one off servers on the top floor of a mid rise office building. I eventually moved it all to a former nuclear fallout facility where our systems were three floors down and under a gigantic pool of water. Error rates and random crashes fell off IMMEDIATELY. I…

Sun’s UltraSPARC II CPU modules were claimed to be particularly susceptible to cosmic rays circa 2000.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#23
post #13
post #4

I have always wondered whether new ideas are seeded by cosmic rays disturbing neurons in the brain..

That'd mean people underground (or under thick layer of concrete, like in skyscrapers) are not creative.

Sounds like we have a hypothesis and a test environment; now we just need a test for creativity and we can start doing science!

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#24

Ahh, the ol' Cisco get-out-of-jail-free card. Headline would be more accurate if it said "30K unidentified bugs blamed on unverifiable phenomenon each year in Japan".

I'm speculating here but it would be more likely that software bugs cause more errors than cosmic rays.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#26
post #10

They have to do two things :-) 1) Use ECC memory 2) Go underground "One experiment measured the soft error rate at the sea level to be 5,950 failures in time (FIT = failures per billion hours) per DRAM chip. When the same test setup was moved to an underground vault, shielded by over 50 feet (15 m) of rock that effectively eliminated all cosmic rays, zero soft errors were recorded.[6] In this test, all other causes o…

A 'Faraday Cage' should also help.

edit: No idea why the downvotes. Faraday cages have long been thought of as a way to protect electrical devices from the electromagnetic waves which can be a result of solar flares.

I even chatted to someone from NASA’s Solar Dynamics Observatory about it in the past.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#27
post #18
post #10

They have to do two things :-) 1) Use ECC memory 2) Go underground "One experiment measured the soft error rate at the sea level to be 5,950 failures in time (FIT = failures per billion hours) per DRAM chip. When the same test setup was moved to an underground vault, shielded by over 50 feet (15 m) of rock that effectively eliminated all cosmic rays, zero soft errors were recorded.[6] In this test, all other causes o…

Polyethylene is supposedly good at blocking cosmic rays. It would be funny to me if the fix was just a drop ceiling full of old grocery bags.

Have a source? That seems like an awesome way to recycle grocery bags.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#28
post #16

Cosmic rays are, or rather were, the reason why manufactures avoid shipping high end cameras by plane, cosmic rays kill pixels in imaging sensors: https://youtu.be/98FZ8C6HneE?t=479 It's also the reason why all ISS footage has dead pixels on it.

Very interesting, so should people who travel with photography equipment take night flights if possible?

Rays hit our planet regardless of Sun, if anything it might block a few. They are caused by ultra-energetic cosmic events, such as star collisions, black hole feeding cycles.

Cosmic rays are just very potent photons, capable of knocking out electrons from an atom (meaning they are ionizing) causing havoc in precision electronics and, well, our DNA.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#29
post #16

Cosmic rays are, or rather were, the reason why manufactures avoid shipping high end cameras by plane, cosmic rays kill pixels in imaging sensors: https://youtu.be/98FZ8C6HneE?t=479 It's also the reason why all ISS footage has dead pixels on it.

Very interesting, so should people who travel with photography equipment take night flights if possible?

Cosmic rays are not exclusively associated with the sun. There are a lot of Galactic Cosmic Rays still. It's pretty much leftovers from supernovae, black hole mergers, neutron stars, accretion disks and such. But yeah, at night would be better.

Edit: Correction, there is very little difference and fewer cosmic rays during the day. Source: https://arxiv.org/pdf/physics/0105005.pdf

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#30
post #18
post #10

They have to do two things :-) 1) Use ECC memory 2) Go underground "One experiment measured the soft error rate at the sea level to be 5,950 failures in time (FIT = failures per billion hours) per DRAM chip. When the same test setup was moved to an underground vault, shielded by over 50 feet (15 m) of rock that effectively eliminated all cosmic rays, zero soft errors were recorded.[6] In this test, all other causes o…

Polyethylene is supposedly good at blocking cosmic rays. It would be funny to me if the fix was just a drop ceiling full of old grocery bags.

Nice, I'm going to line the rooftops with Tyvek now.
Post reply on HN