Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

For a brief period, the Windows kernel tried to deal with gamma rays

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

51–60 of 124 posts

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#51

Post by sevensor »

spullara wrote: post #19

If you don't believe in bit flips, try this! http://dinaburg.org/bitsquatting.html I did that for a bit on cloudfront.net and got dozens of them in a short amount of time.

I used to work in DRAM manufacturing. I believe 100% in bitflips -- we saw them in testing all the time. I also believe the rate at which bitflips are caused by gamma rays is absolutely dwarfed by the rate at which bitflips are caused by manufacturing defects. It's really easy to manufacture a weak cell when you're making billions of tiny capacitors at the same time. SRAM might also see lots of bitflips, but that seems even more likely than DRAM to be driven by manufacturing defects, given the greater complexity of the SRAM cell.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#52

Post by groestl »

tjoff wrote: post #29
spullara wrote: post #19

If you don't believe in bit flips, try this! http://dinaburg.org/bitsquatting.html I did that for a bit on cloudfront.net and got dozens of them in a short amount of time.

Bit flips in network surely is many orders of magnitude worse than in cache/memory bit flips? And I doubt it has to do with gamma rays. Those with poor or long distance DSL do see a big increase in errors. And while hopefully getting rarer now home routers could easily overheat and introduce errors as soon as they saw more than usual traffic. We recently had issues with the power supply for our router and it also cau…

> Bit flips in network surely is many orders of magnitude worse

Yes, but because it's so prevalent that people expect it, they've added checksums on multiple levels, so the network actually performs better. In this particular instance (DNS queries), it's very unlikely that the data was corrupted in transit: "We believe that UDP checksums are effective at preventing 'bitsquat' attacks and other types of errors that occur after a DNS query leaves a DNS resolver and enters the network."

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#53

Post by Nokinside »

datenwolf wrote: post #49

Invalidating the caches is kind of a cringe inducing approach on this (actual) problem. Especially in HPC radiation related single event upsets have become a real problem. If you do the math, all the silicon area devoted to memory (DRAM, caches, registers) adds up, and what you've got is essentially particle detector. Compared to the effective volume of a purpose designed one (ATLAS, CMS, Super Kaminokade, etc.) rath…

Could physicists and astronomers use all this distributed particle detection for research?

Like a app in a phone or desktop that sends reports back with location and time information.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#54

Post by mjevans »

bboreham wrote: post #21
blattimwind wrote: post #10

Earlier quoted context omitted.

Internal data buses and caches are typically ECC'd. In fact, DRAM and the memory bus are most likely the only non-ECC parts in desktop computers.

The way I remember it, when this happened on our Sun E10Ks, the RAM was ECC but the cache was not. Maybe I misremember. It was 20 years ago.

That was my thought process as well, that an early silicon spin / chip wasn't ECC because it would change state frequently and the circuit budget wasn't worth a soft error; but the external RAM might be ECC because a longer running state might not have such a refresh/detection of error.

I agree that these days I'm unaware of any "serious" CPU where the data isn't at least protected by at least a parity bit on chip.

RISC-V : spec mentions ECC, but in a phrasing that makes it clearly optional. https://github.com/riscv/riscv-v-spec/blob/master/v-spec.ado...

"Many processors use error correction codes in the on-chip cache, including the Intel Itanium and Xeon[28] processors, the AMD Athlon, Opteron, all Zen-[29] and Zen+-based[30] processors (EPYC, EPYC Embedded, Ryzen and Ryzen Threadripper), and the DEC Alpha 21264.[23][31]" https://en.wikipedia.org/wiki/ECC_memory#Cache

I prefer to always use ECC and layers of defenses where persistence of data is in question. Jeff Atwood's reference does point to other options, assuming that validation is baked in to the storage process and distributed in such a way as to identify and correct errors across a distributed infrastructure rather than a single system; the distributed nature means it could be more resilient and the validation of data at rest / comparing results is arguably a higher level of integrity than just ECC can provide. https://blog.codinghorror.com/to-ecc-or-not-to-ecc/

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#55

Post by tjoff »

groestl wrote: post #52
tjoff wrote: post #29

Earlier quoted context omitted.

Bit flips in network surely is many orders of magnitude worse than in cache/memory bit flips? And I doubt it has to do with gamma rays. Those with poor or long distance DSL do see a big increase in errors. And while hopefully getting rarer now home routers could easily overheat and introduce errors as soon as they saw more than usual traffic. We recently had issues with the power supply for our router and it also cau…

> Bit flips in network surely is many orders of magnitude worse Yes, but because it's so prevalent that people expect it, they've added checksums on multiple levels, so the network actually performs better. In this particular instance (DNS queries), it's very unlikely that the data was corrupted in transit: "We believe that UDP checksums are effective at preventing 'bitsquat' attacks and other types of errors that oc…

That doesn't take into consideration of flaky home-routers (note that they do recalculate the checksum). So the checksum might be processed on bad data.

For a bad connection it will also be quite common that the checksum will happen to be valid. TCP/UDP checksums are quite bad.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#56

Post by ajb »

Nokinside wrote: post #53
datenwolf wrote: post #49

Invalidating the caches is kind of a cringe inducing approach on this (actual) problem. Especially in HPC radiation related single event upsets have become a real problem. If you do the math, all the silicon area devoted to memory (DRAM, caches, registers) adds up, and what you've got is essentially particle detector. Compared to the effective volume of a purpose designed one (ATLAS, CMS, Super Kaminokade, etc.) rath…

Could physicists and astronomers use all this distributed particle detection for research? Like a app in a phone or desktop that sends reports back with location and time information.

Most bit flip events are apparently due to alpha particles from radioactive decay in the package (source - worked for a company which used 'low alpha compound' in some of our ICs to put off having to implement HW error correction) , which would be a big confounding signal. I imaging physics experiments work hard to avoid this, or at least correct for it.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#57

Post by richardlblair »

spullara wrote: post #19

If you don't believe in bit flips, try this! http://dinaburg.org/bitsquatting.html I did that for a bit on cloudfront.net and got dozens of them in a short amount of time.

I was talking to a guy at Github who had to debug an enterprise deployment that just stopped working one day.

There was a bit that flipped and apparently took the whole thing to its knees.

This is the stuff nightmares are made of.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#58

Post by myrandomcomment »

A bunch of years ago Cisco had an issue with some RAM in a new switch model, I think it was in the 65xx. They where crashing randomly but only in certain places in the world. Cisco spent tons of money on this. No idea. They brought in a physics professor. The devices with the most issues were located in countries up near the artic circle. Cosmic Rays caused a bit flip in this particular set of RAM due to something in its design. Sorry for the light details, it's been years.

I also worked at a switch manufacturer. We had some ASICs from one of the big companies. Had crashes that we could not explain at all. We knew it was not us. Proved that bits where flipping in the switch ASIC. Turn out they had forgot to spec low alpha solder. Alpha partics will not go through your skin, but when it is layered right on to the chip....oops.

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#59

Post by Faaak »

spullara wrote: post #19

If you don't believe in bit flips, try this! http://dinaburg.org/bitsquatting.html I did that for a bit on cloudfront.net and got dozens of them in a short amount of time.

Did also that with facebook.fr/google.fr/akamai.net domains. Got a _lot_ of hits, with mostly mobile user agents.

Was fun; never published my results though..

Re: For a brief period, the Windows kernel tried to deal with gamma rays

#60

Post by nategri »

21 wrote: post #27
nategri wrote: post #15

A pedantic point (good thing I'm on HN), but I wonder if they didn't actually mean muons and not gamma rays?

Apparently 95% of "bit flipping cosmic rays" are neutrons: > At the Earth's surface approximately 95% of the particles capable of causing soft errors are energetic neutrons with the remainder composed of protons and pions. https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creatin...

Great find. And somewhat surprising because hadrons mostly don't survive to sea level. Too much integrated cross-section with the atmosphere.
Post reply on HN