Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

Post reply on HN
Print view View original

21–30 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#21

Post by dijit »

dannyw wrote: post #17

Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

These are all very fair statements but there’s no guarantee that ECC memory was even used. Computers typically fail open when ECC is potentially present but not available.

People also cite early stage google and intentionally do not buy ECC components, running more consumer hardware for production workloads.

Even if google later recanted that theology.

Re: Single random bit flip causes error in certificate transparency log

#22

Post by numpad0 »

bloopernova wrote: post #4

Layman question: Is it possible to shield electronics against cosmic rays? Or is it like a neutrino thing, where they just go through everything? (But neutrinos barely interact, correct?)

Not trivially; if it was viable the Chernobyl reactor would be gone already. Also, IC packagings and chassis materials can be sources of stray electron beams, and other modes of glitching like power failure exists.

Re: Single random bit flip causes error in certificate transparency log

#24

Post by juped »

gbrown_ wrote: post #10

Can't say I really like the title as it comes across as an absolute statement whereas the bit flip could have happened for any number of unknown reasons. I saw a similar take on Twitter and whilst root causing such things (especially when it's a single occurrence) isn't always possible, shrugging and saying "cosmic rays" should be the last thing to posit not one of the first.

"cosmic rays" are more of a well-known term of art for "single bit flipped with unknown hardware cause" than a reference to literal cosmic rays

Re: Single random bit flip causes error in certificate transparency log

#25

Post by caspper69 »

dijit wrote: post #21
dannyw wrote: post #17

Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

These are all very fair statements but there’s no guarantee that ECC memory was even used. Computers typically fail open when ECC is potentially present but not available. People also cite early stage google and intentionally do not buy ECC components, running more consumer hardware for production workloads. Even if google later recanted that theology.

It’s always humorous to me when people use the term theology in situations such as this; it makes me wonder, as human mental bandwidth becomes more strained and we increase our specializations to the n-th degree, what will constitute theology in the future?

Food for thought.

Re: Single random bit flip causes error in certificate transparency log

#27

Post by gbrown_ »

juped wrote: post #24
gbrown_ wrote: post #10

Can't say I really like the title as it comes across as an absolute statement whereas the bit flip could have happened for any number of unknown reasons. I saw a similar take on Twitter and whilst root causing such things (especially when it's a single occurrence) isn't always possible, shrugging and saying "cosmic rays" should be the last thing to posit not one of the first.

"cosmic rays" are more of a well-known term of art for "single bit flipped with unknown hardware cause" than a reference to literal cosmic rays

It is not a "well-known term", it is a cliché.

Re: Single random bit flip causes error in certificate transparency log

#28

Post by kzrdude »

layoutIfNeeded wrote: post #2

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

I'm just thinking (of course) that you said If the bit-flip had occured but it's probably already when the bit flip occurs in the .TEXT section; we don't know what it might already have caused or just passed without notice (unreproducible bug, or bitflip in function that's never called or whatever).

Re: Single random bit flip causes error in certificate transparency log

#29

Post by Denvercoder9 »

henearkr wrote: post #23

Bit flips like that should be easy to reverse. Flip just one bit at the n-th place and test again the certificate, and vary n, until it is valid. It's done in linear time.

There's a bit flip in the recorded hash of the certificate, not in the certificate itself. You'd need to break SHA-2 to reverse that.

Re: Single random bit flip causes error in certificate transparency log

#30

Post by fulafel »

dannyw wrote: post #17

Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

These things exist, using trade names like chipkill or lockstep memory. Though they don't need to sacrifice half of the memory chips to get good error recovery properties.

Note that this is still not end-to-end protection of data integrity. Bit flips happen in networking, storage, buses between everything, caches, CPUs, etc. See eg [1]

[1] https://arxiv.org/abs/2102.11245 Silent Data Corruptions at Scale (based on empirical data at Facebook)

Post reply on HN