Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

Post reply on HN
Print view View original

41–50 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#41

Post by Leparamour »

Kimitri wrote: post #14
superjan wrote: post #3

Earlier quoted context omitted.

I am used to ‘bit rot’ refer to code becoming obsolete due to lack of maintenance. Can’t we use another term for actual hardware errors?

It may be a regional thing but I have never heard ”bit rot” refer to legacy code. In the retro computing circles bit rot refers to hardware defects (usually floppies or other storage media) caused by cosmic rays or other environmental hazards.

I have to agree with Kimitri here. This is the only context in which I have ever encountered the term 'bit rot'.

Re: Single random bit flip causes error in certificate transparency log

#42

Post by solarkraft »

juped wrote: post #24
gbrown_ wrote: post #10

Can't say I really like the title as it comes across as an absolute statement whereas the bit flip could have happened for any number of unknown reasons. I saw a similar take on Twitter and whilst root causing such things (especially when it's a single occurrence) isn't always possible, shrugging and saying "cosmic rays" should be the last thing to posit not one of the first.

"cosmic rays" are more of a well-known term of art for "single bit flipped with unknown hardware cause" than a reference to literal cosmic rays

Interesting, because I was thinking of ... cosmic rays.

Re: Single random bit flip causes error in certificate transparency log

#43

Post by formerly_proven »

fulafel wrote: post #30
dannyw wrote: post #17

Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

These things exist, using trade names like chipkill or lockstep memory. Though they don't need to sacrifice half of the memory chips to get good error recovery properties. Note that this is still not end-to-end protection of data integrity. Bit flips happen in networking, storage, buses between everything, caches, CPUs, etc. See eg [1] [1] https://arxiv.org/abs/2102.11245 Silent Data Corruptions at Scale (based on em…

According to the Intel developer's manual L1 has parity and all caches up from that have ECC. This would seem to imply that the ring / mesh also has at least parity (to retry on error). Parity instead of ECC on L1(D) makes sense since the L1(D) has to handle small writes well, while the other caches deal in lines.

Re: Single random bit flip causes error in certificate transparency log

#46

Post by Tomte »

bloopernova wrote: post #4

Layman question: Is it possible to shield electronics against cosmic rays? Or is it like a neutrino thing, where they just go through everything? (But neutrinos barely interact, correct?)

No, because bit flips from radioactive decay are not typically cosmic rays, but originating in the chip's packaging.

Re: Single random bit flip causes error in certificate transparency log

#47

Post by ben_w »

Leparamour wrote: post #38
noduerme wrote: post #11

Earlier quoted context omitted.

That's not what actually happened here (probably), but think of relatively heavy atoms like iron, ejected from a supernova at close to the speed of light. They're not really "rays". They're solid particles that punch holes in everything. The good news is, mostly they're ionized and they get repelled away from hitting the planet's surface by our magnetic field (generated by the big hot iron magnet that's churning unde…

> That's not what actually happened here (probably), but think of relatively heavy atoms like iron, ejected from a supernova at close to the speed of light. They're not really "rays". Wow, I have to admit I always assumed cosmic rays to be gamma radiation but alpha particle radiation sounds a lot more scary. Does anyone happen to know if computers engineered for the space station or shuttles have already some built-i…

Radiation hardening is basically everything you can manage with the weight limit: https://en.wikipedia.org/wiki/Radiation_hardening

IIRC nobody is currently using magnetic fields for shielding, I don’t know if that’s due to insufficient effectiveness, power consumption, or unwanted interactions e.g. with Earth’s magnetosphere.

Re: Single random bit flip causes error in certificate transparency log

#48

Post by nicoburns »

bloopernova wrote: post #4

Layman question: Is it possible to shield electronics against cosmic rays? Or is it like a neutrino thing, where they just go through everything? (But neutrinos barely interact, correct?)

Yes, the easiest way is to place the computer underground (or underwater).

Re: Single random bit flip causes error in certificate transparency log

#49

Post by fmajid »

layoutIfNeeded wrote: post #2

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

I’ve had a case where a bit flip in a TCP stream was not caught because it happened in a Singapore government deep packet inspection snoop gateway that recalculated the TCP checksum for the bit-flipped segment:

https://blog.majid.info/telco-snooping/

Re: Single random bit flip causes error in certificate transparency log

#50

Post by jacquesm »

tsegratis wrote: post #45

Do nuclear missiles have this anywhere? bool launch = false; if(launch) ...

It doesn't really matter whether they have

    if (launch1 && launch2 && launch3) {
        launch();
    }
either because a single bit flip (of the code) could still cause a launch. You'd hope that it was at least stored in ROM and that ECC ensures that even if such a bit flips it does not lead straight to Armageddon.

There are some 'near miss' stories where a single switch made all the difference:

https://www.theatlantic.com/technology/archive/2013/09/the-s...

So I would not be all that surprised if there are equivalent single bits.

Post reply on HN