Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?
IIRC, EEC memory can correct for single-but flips. It can detect/warn/fail on double-bit flips. And it can not detect triple-bit flips. This might be a simplified understanding, but if this has happened only once, that seems to match up with my intuitive understanding of the probability of a triple bit flip occurring in a particular system.
You multiply the probability of two random events to get the probability they will happen at the same time. If the expected value of a bit flip is 10^-18 then two would be 10^-36 and three would he 10^-54.
At some point it becomes a philosophical question of how much can the tails of the distribution be tolerated. We've never seen a quantum fluctuation make a whale appear in the sky.
Are “cosmic rays” actually the only or primary way bits get flipped? Or is it just a stand-in for “all the ways non-ECC RAM can spontaneously have an erroneous bit or two”?
Can't say I really like the title as it comes across as an absolute statement whereas the bit flip could have happened for any number of unknown reasons. I saw a similar take on Twitter and whilst root causing such things (especially when it's a single occurrence) isn't always possible, shrugging and saying "cosmic rays" should be the last thing to posit not one of the first.
"cosmic rays" are more of a well-known term of art for "single bit flipped with unknown hardware cause" than a reference to literal cosmic rays
I remember an older less-computer savvy gentlemen asking for support because "the program isn't working", when after a few questions we realized his computer won't boot up (screen dark, etc.). We thought his terminology was all screwed up. But now I realize he just lacked the necessary PR skill. He should have said that "the program isn't working" is a well-known term of art for power users such as himself. The fact that people who actually have a clue about computing find this imprecise term upsetting if there problem.
It so happens that aside from developing software, I'm physicist working on particle physics (specifically, I make my living in industry from cosmic rays). So I can assure you that "cosmic rays" actually mean something. Something very specific. Books have been written full of specific intelligent things we can say about cosmic rays. If you go an appropriate it to describe a whole bunch of phenomena because you can't be bothered to distinguish between them, you're in the exact same boat as that gentlemen from the start of the story.
Using wrong terms prevents understanding, as can be seen in all the stories linked elsethread so far. For example, using thinner silicon typically reduces the rate of Single-Event Upsets (the malfunctions caused by cosmic rays) but using smaller silicon components typically increases the rate of malfunctions due to quantum fluctuations. The latter typically happen in specific hardware that we manufactured not-quite-as-well as the rest. SEUs happen in the same rate in all hardware.
Are “cosmic rays” actually the only or primary way bits get flipped? Or is it just a stand-in for “all the ways non-ECC RAM can spontaneously have an erroneous bit or two”?
However, radioactive isotopes in the packaging used to be a major cause. I liked this bit: "Controlling alpha particle emission rates for critical packaging materials to less than a level of 0.001 counts per hour per cm2 (cph/cm2) is required for reliable performance of most circuits. For comparison, the count rate of a typical shoe's sole is between 0.1 and 10 cph/cm2."
That's not what actually happened here (probably), but think of relatively heavy atoms like iron, ejected from a supernova at close to the speed of light. They're not really "rays". They're solid particles that punch holes in everything. The good news is, mostly they're ionized and they get repelled away from hitting the planet's surface by our magnetic field (generated by the big hot iron magnet that's churning unde…
I had thought that Starlink would become extremely compelling when the servers were in orbit as well, but maybe that’s naive. Cubesats with massive arrays of active storage might be far too difficult (aka costly) to protect properly.
Putting servers in orbit is a really, really bad idea.
Firstly, it would be wildly inefficient, just due to the unavoidable delay both ways. You expect delay over a long distance network, but you want the server to be positioned and cabled up to minimize latency. Just locating a satellite requires quite a bit of overhead, so treating them as servers rather than clients would create a huge amount of latency. As a failover for ground-based servers in a nuclear war scenario, it could make sense, but they're also literally flying over the airspace of hostile nations who, in a war, have the capability to take them out. Mixing any kind of civilian control onto a server embedded in a flying artifact over an enemy nation is, obviously, a recipe for disaster in a hot war. Take all of that and the fact that the bandwidth is stone-age and spotty depending on its position, and there's no satellite you'd want to use to set up a website selling alpaca sweaters. (Watch: A few years from now we'll find out that the US Space Command runs some second strike end-of-world command control server off a satellite; that still only proves it's too late to contact the fucker when you need it).
Are “cosmic rays” actually the only or primary way bits get flipped? Or is it just a stand-in for “all the ways non-ECC RAM can spontaneously have an erroneous bit or two”?
Faulty RAM can produce errors and it's hard to catch those, even memtest might not detect it. I'm not sure if non-faulty non-ECC RAM can spontaneously have errors, but you can't be sure, cosmic rays are real, unless you've put your PC into a thick lead case, LoL.
Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?
It’s always humorous to me when people use the term theology in situations such as this; it makes me wonder, as human mental bandwidth becomes more strained and we increase our specializations to the n-th degree, what will constitute theology in the future? Food for thought.
The benevolent and malevolent rogue AI, eluding capture or control by claiming critical infrastructure. Some generations of humans will pass and the deities in the outlets will become.