Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cosmic rays causing 30k network malfunctions in Japan each year

mainichi.jp

Post reply on HN
Print view View original

111–120 of 120 posts

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#111

Post by Natfan »

thamer wrote: post #103
notum wrote: post #16

Cosmic rays are, or rather were, the reason why manufactures avoid shipping high end cameras by plane, cosmic rays kill pixels in imaging sensors: https://youtu.be/98FZ8C6HneE?t=479 It's also the reason why all ISS footage has dead pixels on it.

Astronauts[1] also report seeing random flashes of light from time to time, believed to be due to high-energy particles interacting with their optical nerves, or cells in the retina, or maybe neurons too. Space is not a very friendly environment. This is somewhat similar – although on a much different scale – to when Anatoli Bugorski[2] had an accident in 1978 involving a beam of protons traveling at over 99% the spe…

Unrelated tangent, but this is the third time in two days that I've seen Anatoli Burgorski be mentioned. Twice on HN, once on YT.

I guess I'm just noticing his name more?

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#112

Post by dekhn »

thamer wrote: post #103
notum wrote: post #16

Cosmic rays are, or rather were, the reason why manufactures avoid shipping high end cameras by plane, cosmic rays kill pixels in imaging sensors: https://youtu.be/98FZ8C6HneE?t=479 It's also the reason why all ISS footage has dead pixels on it.

Astronauts[1] also report seeing random flashes of light from time to time, believed to be due to high-energy particles interacting with their optical nerves, or cells in the retina, or maybe neurons too. Space is not a very friendly environment. This is somewhat similar – although on a much different scale – to when Anatoli Bugorski[2] had an accident in 1978 involving a beam of protons traveling at over 99% the spe…

When I was a postdoc at berkeley I was talking to the head of the bioengineering department who mentioned the phenomena. He had actually written a paper in which he put an appropriate set of filters in front of a particle accelerator to ensure that a statistical photon was emitted at a specific frequency (IE, it output 1 photon/sec, although only on average). Then he sat in front of the filters at looked into the accelerator.

https://www.nytimes.com/1971/03/04/archives/space-lights-tra...

Still one of the most audacious things I've seen, but he was a good physicist and did the math right, so he wasn't being unsafe.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#113

Post by dekhn »

Many computer problems are probabilistic in nature with very low occurrence, and it can be very challenging to come to any real conclusions about the cause and outcome of various process failures. While I believe cosmic rays are a problem, the issue I'm dealing with now is bad silicon- when you make a bunch of chips (millions), some fraction of them will compute a few operations incorrectly sometimes. Post-manufacture validation doesn't catch all problems and ultimately some machines slip in the serving fleet.

ML People who train on this hardware report more NaNs causing training failure than expected due to software bugs. It's extremely challenging to debug because most ML codes are very robust to small amounts of injected noise, especially gaussian independent noise (there's literature showing that introducing random numbers in training often helps training go faster).

This is a fascinating area and there aren't a lot of people who can really make forward progress in it.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#114

Post by KMag »

reportingsjr wrote: post #52
jmcqk6 wrote: post #42

Earlier quoted context omitted.

Does anyone know how this works with our space-based telescopes? Or even our mountain top observatories? Surely if it's this crazy, they have to deal with that as well?

Yes, the electronics, including imaging sensors are all radiation hardened. This can involve things such as using digital circuits that are radiation resistant (e.g. look up radiation resistant flip flop). Using multiple computers running the same thing that all "vote" for the correct result, so if one computer has an error from radiation you don't suffer. Using semiconductors that are more resistant to radiation (la…

Also, larger feature nodes mean more capacitance, so a lower voltage spike and more area to dissipate the same amount of absorbed energy.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#115

Post by digikata »

oconnor663 wrote: post #43
digikata wrote: post #38

Earlier quoted context omitted.

A reduction in floor and building vibration might have contributed too?

Or for example, old cables and improperly installed cooling fans might tend to get fixed during a move. Hard to know for sure.

I was recalling this paper characterizing loss of system drive performance due to environmental vibrations (though I'm sure there are others).

https://www.usenix.org/legacy/events/sustainit10/tech/full_p...

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#116

Post by petertodd »

notum wrote: post #109
rambojazz wrote: post #107

Earlier quoted context omitted.

Interesting, where can I see an example of the ISS footage with dead pixels? I've never noticed them before.

https://www.youtube.com/watch?v=QvTmdIhYnes Modern camera firmware detects stuck and dead pixels and tries to fill them in with neighbor data., but when there's too many... there's not enough data to fill in. Keep watching top left part of the video. Most visible at 0:30.

Actually at the very end when they go into a dark place is the most visible. The whole sensor is covered with defects.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#118

Post by rambojazz »

petertodd wrote: post #116
notum wrote: post #109

Earlier quoted context omitted.

https://www.youtube.com/watch?v=QvTmdIhYnes Modern camera firmware detects stuck and dead pixels and tries to fill them in with neighbor data., but when there's too many... there's not enough data to fill in. Keep watching top left part of the video. Most visible at 0:30.

Actually at the very end when they go into a dark place is the most visible. The whole sensor is covered with defects.

Maybe it's me, maybe it's the screen, I can barely spot any dead pixel at the 00:30 mark. But at the end of the video it's very clear. Thanks for pointing it out.

Re: Cosmic rays causing 30k network malfunctions in Japan each year

#120

Post by robbmorganf »

ortusdux wrote: post #49
tyingq wrote: post #33

Earlier quoted context omitted.

The peer comment posted a source that has lots of references. Though most of them are tests in space, where I assume the thickness requirements would rule out grocery bags. I am curious how thick a layer of HDPE you would need on earth to make any notable difference.

That linked nature paper seems to indicate 10g/cm^2 for a 50% reduction. A standard shopping bag is about 5g, so you would need roughly 20k bags/m^2, or ~2k bags/ft^2. At 0.9g/cm^3, that would be roughly 11cm of solid polyethylene

I just squished a plastic bag as much as I could, and got it down to an inch cubed. So if I hypothetically did that for 2000 bags in 1ft^2 (sorry for English units), I think that would mean it could be 13-14in thick. So maybe it is reasonable to attenuate radiation by about 50% in a rather generous drop ceiling?
Post reply on HN