Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For a brief period, the Windows kernel tried to deal with gamma rays

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

81–90 of 124 posts

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#81

Post by vvanders »

russdill wrote: post #79
weberc2 wrote: post #78

When I was fresh out of college, I worked as a contractor for a prominent agricultural equipment manufacturer. I was responsible for building out the touch-screen interface for the radio (a Qt app). I was told by an engineer who worked for the equipment manufacturer that my application wasn't good enough because needed to be able to operate correctly in the face of arbitrary bit flips "from lightning strikes"--I kind…

The requirement request is a great way to push back on feature creep. There's a lot of cargo culting that goes on in the "protection against bit-flips". You sometimes have to go a step further and ask what error rate are you required to be below. Once you have that number, you can start asking what your current error rate is without mitigations, and how much a given mitigation will reduce your error rate.

My favorite entry in that problem space is metastability[1].

Do you interface two different clock domains(which is basically most things)? Guess what all of your computing is built on the "chance" that bits won't flip.

Granted, statistics make this pretty solid but kinda blew my mind when I first stumbled across it.

[1] https://en.wikipedia.org/wiki/Metastability_(electronics)

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#82

Post by russdill »

myrandomcomment wrote: post #58

A bunch of years ago Cisco had an issue with some RAM in a new switch model, I think it was in the 65xx. They where crashing randomly but only in certain places in the world. Cisco spent tons of money on this. No idea. They brought in a physics professor. The devices with the most issues were located in countries up near the artic circle. Cosmic Rays caused a bit flip in this particular set of RAM due to something in…

Xilinx tracks a ton of data in regard to their products, packaging, and bit upsets:

https://forums.xilinx.com/t5/PLD-Blog-Archived/Package-Gener...

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#83

Post by russdill »

blattimwind wrote: post #13
fpgaminer wrote: post #11

So ... here's what I'm thinking, as a complete layman with respect to how radiation affects memory devices. RAM is DRAM, i.e. dynamic RAM. It has to get automatically refreshed relatively frequently. So, maybe (again, me being a layman) what happens is that usually gamma rays hit a DRAM cell, but haven't imparted enough energy to cause a flip. A millisecond later the cell gets refreshed erasing what little influence…

> Contrast this with processor cache. On-die cache is most likely SRAM, Static RAM. It doesn't get refreshed. So the slight voltage errors caused by gamma rays can slowly build up over time. Static RAM is basically a flip-flop. It's a bistable circuit that's actively held in a stable state. Single-event upsets work by, essentially, putting the energy into the circuit required to make it transition into the opposite s…

In layman's terms, it's being continuously refreshed.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#84

Post by russdill »

dekhn wrote: post #2

If you have a large enough fleet, and log your ECC errors, you have actually built a not-very-sensitive and very expensive scientific instrument- a cosmic ray detector. Physics is awesome.

As long as you aren't high altitude or near the poles, I think most of your upset events will be from trace radioactive elements in your packaging.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#85

Post by russdill »

vvanders wrote: post #81
russdill wrote: post #79

Earlier quoted context omitted.

The requirement request is a great way to push back on feature creep. There's a lot of cargo culting that goes on in the "protection against bit-flips". You sometimes have to go a step further and ask what error rate are you required to be below. Once you have that number, you can start asking what your current error rate is without mitigations, and how much a given mitigation will reduce your error rate.

My favorite entry in that problem space is metastability[1]. Do you interface two different clock domains(which is basically most things)? Guess what all of your computing is built on the "chance" that bits won't flip. Granted, statistics make this pretty solid but kinda blew my mind when I first stumbled across it. [1] https://en.wikipedia.org/wiki/Metastability_(electronics)

Yup, a large portion of hardware design is based on getting below a required maximum failure rate. For metastability, you just keep adding more flip-flops. BTW, the cache invalidation request may be due to this. They figured they could more easily reach their time between failure interval if they could discount time during S1.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#86

Post by drharby »

vvanders wrote: post #81
russdill wrote: post #79

Earlier quoted context omitted.

The requirement request is a great way to push back on feature creep. There's a lot of cargo culting that goes on in the "protection against bit-flips". You sometimes have to go a step further and ask what error rate are you required to be below. Once you have that number, you can start asking what your current error rate is without mitigations, and how much a given mitigation will reduce your error rate.

My favorite entry in that problem space is metastability[1]. Do you interface two different clock domains(which is basically most things)? Guess what all of your computing is built on the "chance" that bits won't flip. Granted, statistics make this pretty solid but kinda blew my mind when I first stumbled across it. [1] https://en.wikipedia.org/wiki/Metastability_(electronics)

Hurray digital signal synchronization!

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#87

Post by dfox »

groestl wrote: post #52
tjoff wrote: post #29

Earlier quoted context omitted.

Bit flips in network surely is many orders of magnitude worse than in cache/memory bit flips? And I doubt it has to do with gamma rays. Those with poor or long distance DSL do see a big increase in errors. And while hopefully getting rarer now home routers could easily overheat and introduce errors as soon as they saw more than usual traffic. We recently had issues with the power supply for our router and it also cau…

> Bit flips in network surely is many orders of magnitude worse Yes, but because it's so prevalent that people expect it, they've added checksums on multiple levels, so the network actually performs better. In this particular instance (DNS queries), it's very unlikely that the data was corrupted in transit: "We believe that UDP checksums are effective at preventing 'bitsquat' attacks and other types of errors that oc…

One thing to keep in mind is that the network protocols usually use quite weak and similar checksums. To the extent that there are layer combinations that reliably produce errors that are not detectable by TCP checksum.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#88

Post by skunkworker »

datenwolf wrote: post #49

Invalidating the caches is kind of a cringe inducing approach on this (actual) problem. Especially in HPC radiation related single event upsets have become a real problem. If you do the math, all the silicon area devoted to memory (DRAM, caches, registers) adds up, and what you've got is essentially particle detector. Compared to the effective volume of a purpose designed one (ATLAS, CMS, Super Kaminokade, etc.) rath…

An article I read awhile ago addressed an interesting correlation between transistor process size, the physical size of the dram module and the expected failure rates.

"As transistor sizes have shrunk, they have required less and less electrical charge to represent a logical bit. So the likelihood that one bit will "flip" from 0 to 1 (or 1 to 0) when struck by an energetic particle has been increasing. This has been partially offset by the fact that as the transistors have gotten smaller they have become smaller targets so the rate at which they are struck has decreased.

More significantly, the current generation of 16-nanometer circuits have a 3D architecture that replaced the previous 2D architecture and has proven to be significantly less susceptible to SEUs. Although this improvement has been offset by the increase in the number of transistors in each chip, the failure rate at the chip level has also dropped slightly. However, the increase in the total number of transistors being used in new electronic systems has meant that the SEU failure rate at the device level has continued to rise."

[1] https://phys.org/news/2017-02-particles-outer-space-wreaking...

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#89

Post by ChuckMcM »

datenwolf wrote: post #49

Invalidating the caches is kind of a cringe inducing approach on this (actual) problem. Especially in HPC radiation related single event upsets have become a real problem. If you do the math, all the silicon area devoted to memory (DRAM, caches, registers) adds up, and what you've got is essentially particle detector. Compared to the effective volume of a purpose designed one (ATLAS, CMS, Super Kaminokade, etc.) rath…

And vendors who put ECC on the data paths but not on the cache itself, on the assumption that check it while it is going in, its a cache so it is short lived, and you're window is small enough to meet your reliability goals.

That goes out the window when you sleep though because you don't know how long you've been waiting to restart. And the P(bitflip) is a function of time. If you sleep too long you are non-spec compliant for silent data corruption, and since there isn't a way to know how long you sleep, the only "safe" option is to invalidate the cache and reload it.

Sad but an understandable approach. The downside is your wake from sleep is slower by the amount of time it takes to warm up the cache.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#90

Post by toomanybeersies »

trhway wrote: post #71
sevensor wrote: post #51

Earlier quoted context omitted.

I used to work in DRAM manufacturing. I believe 100% in bitflips -- we saw them in testing all the time. I also believe the rate at which bitflips are caused by gamma rays is absolutely dwarfed by the rate at which bitflips are caused by manufacturing defects. It's really easy to manufacture a weak cell when you're making billions of tiny capacitors at the same time. SRAM might also see lots of bitflips, but that see…

>the rate at which bitflips are caused by gamma rays is absolutely dwarfed by the rate at which bitflips are caused by manufacturing defects. back in USSR/Russia gamma rays and aliens were not-an-issue compare to the extremely low reliability of USSR/Russia hardware. The military hardware back then (and some telco hardware built in Russia in 199x (some even for export into Western countries!)) was built as triplicate…

Except for the case in the USSR when gamma rays were an issue at a train depot computer system due to radioactive cows from near Chernobyl: http://www.jakepoz.com/debugging-behind-the-iron-curtain/
Post reply on HN