Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

11–20 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#11

Layman question: Is it possible to shield electronics against cosmic rays? Or is it like a neutrino thing, where they just go through everything? (But neutrinos barely interact, correct?)

That's not what actually happened here (probably), but think of relatively heavy atoms like iron, ejected from a supernova at close to the speed of light. They're not really "rays". They're solid particles that punch holes in everything. The good news is, mostly they're ionized and they get repelled away from hitting the planet's surface by our magnetic field (generated by the big hot iron magnet that's churning under our crust). But in the rest of space where there's no field like that, getting lots of tiny puncture wounds from high-velocity iron particles is pretty normal. It does a lot of damage to cells and DNA in a cumulative way pretty quickly.

To answer your question, shielding is possible but it is much harder in space than it is under the magnetosphere. Even so, a stray particle can wreak havoc. Shielding for the purposes of data protection on earth is essentially a risk/reward analysis. Anyone could get hit by a cosmic bullet, but the chances are pretty low. The odds of an SSD flipping bits on its own are significantly higher.

Re: Single random bit flip causes error in certificate transparency log

#12
post #3

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

I am used to ‘bit rot’ refer to code becoming obsolete due to lack of maintenance. Can’t we use another term for actual hardware errors?

"sharting"

Re: Single random bit flip causes error in certificate transparency log

#13

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

It's also an excuse/opportunity to write a fairly Hard SF zombie story:

https://www.antipope.org/charlie/blog-static/bit-rot.html

Re: Single random bit flip causes error in certificate transparency log

#14
post #3

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

I am used to ‘bit rot’ refer to code becoming obsolete due to lack of maintenance. Can’t we use another term for actual hardware errors?

It may be a regional thing but I have never heard ”bit rot” refer to legacy code. In the retro computing circles bit rot refers to hardware defects (usually floppies or other storage media) caused by cosmic rays or other environmental hazards.

Re: Single random bit flip causes error in certificate transparency log

#15
post #5

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

Were you actually able to diff it down to a single bit flip?

It's just a story we usually tell the boss when the whole team wants to goof off for a few days.

Re: Single random bit flip causes error in certificate transparency log

#16

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

And that's why we need deterministic builds

Re: Single random bit flip causes error in certificate transparency log

#17
Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips?

As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache?

Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

Re: Single random bit flip causes error in certificate transparency log

#18
post #5

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

Were you actually able to diff it down to a single bit flip?

Yes. Via git diff --no-index and xxd.

Re: Single random bit flip causes error in certificate transparency log

#19
post #6

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

Is this bit rot happening on a server w/ ECC RAM just curious?

It was a MacStadium (https://www.macstadium.com) build server, so most likely non-ECC.

Re: Single random bit flip causes error in certificate transparency log

#20

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

And that's why we need deterministic builds

Yes, but in this case this was a bit-flip in a third-party binary dependency that we don't have the source code for.
Post reply on HN