Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

61–70 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#61
post #52
post #3

Earlier quoted context omitted.

I am used to ‘bit rot’ refer to code becoming obsolete due to lack of maintenance. Can’t we use another term for actual hardware errors?

I’ve always understood “bit rot” meaning data getting silently corrupted on o storage device like a hard drive or SSD.

Same here. “Bit rot” is then analogous to food rot: the longer your data sits unverified, the more likely that there will be flipped bits and therefore “rotten data”.

Re: Single random bit flip causes error in certificate transparency log

#62
post #6

Earlier quoted context omitted.

Is this bit rot happening on a server w/ ECC RAM just curious?

It was a MacStadium ( https://www.macstadium.com ) build server, so most likely non-ECC.

Strange that build services are not just going full ECC, especially with cheaper hardware now supporting it.

Re: Single random bit flip causes error in certificate transparency log

#63

Do nuclear missiles have this anywhere? bool launch = false; if(launch) ...

There have been false alerts caused by single chip failures. 1980 there was nuclear alert in the US that lasted over three minutes.

Generally critical systems can't be armed without physical interaction from humans. It's not just computer logic, but powering up the system that can do the launch. It does not matter what the logic does as long as ignition system is not powered up using physical switch.

Re: Single random bit flip causes error in certificate transparency log

#64

Earlier quoted context omitted.

Yes. Via git diff --no-index and xxd.

I have no words. Stories like this trigger PTSD for me. I wasn't trying to be flip. That must have been a bitch to figure out.

Well, it wasn't that hard to uncover actually. We knew that the same build succeeds on our machines. So we only had to find what the difference was between the two :)

As Arthur Conan Doyle put it: "Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth." ¯\_(ツ)_/¯

Re: Single random bit flip causes error in certificate transparency log

#65
The push for crypto without ecc ram is a nonstop horror show. Software under normal circumstances is remarkably resilient to having its memory corrupted. However crypto algorithms are designed so that a single bit flip effectively changes all the bits in a block. If you chain blocks then a single bit flip in one block destroys all the blocks. I've seen companies like msps go out of business because they were doing crypto with consumer hardware. No one thinks it'll happen to them and once it happens they're usually too dim to even know what happened. Bit flips aren't an act of god you simply need a better computer.

Re: Single random bit flip causes error in certificate transparency log

#66
post #30

Earlier quoted context omitted.

These things exist, using trade names like chipkill or lockstep memory. Though they don't need to sacrifice half of the memory chips to get good error recovery properties. Note that this is still not end-to-end protection of data integrity. Bit flips happen in networking, storage, buses between everything, caches, CPUs, etc. See eg [1] [1] https://arxiv.org/abs/2102.11245 Silent Data Corruptions at Scale (based on em…

According to the Intel developer's manual L1 has parity and all caches up from that have ECC. This would seem to imply that the ring / mesh also has at least parity (to retry on error). Parity instead of ECC on L1(D) makes sense since the L1(D) has to handle small writes well, while the other caches deal in lines.

Of course, parity can only detect an odd number of bit flips.

Re: Single random bit flip causes error in certificate transparency log

#67
Cosmic-ray bit flipping is real and it has real security concerns. This also makes Intel's efforts at market segmentation by not having ECC support in any consumer CPUs [1] even more unforgivable and dangerous.

Example: bitsquatting on domains [2].

[1]: https://arstechnica.com/gadgets/2021/01/linus-torvalds-blame...

[2]: https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...

Re: Single random bit flip causes error in certificate transparency log

#68
post #21
post #17

Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

These are all very fair statements but there’s no guarantee that ECC memory was even used. Computers typically fail open when ECC is potentially present but not available. People also cite early stage google and intentionally do not buy ECC components, running more consumer hardware for production workloads. Even if google later recanted that theology.

Public CAs are not that type of people; I would be disapointed if that were not running two seperate systems checking each other for consistancy; having top of the range ECC running well inside its specification must be table stakes.

Re: Single random bit flip causes error in certificate transparency log

#70

Do nuclear missiles have this anywhere? bool launch = false; if(launch) ...

I don't know about nuclear missiles, but on the Space Shuttle I think they had four duplicate flight computers, and the outputs of all of them would be compared to look for errors. (They also had a fifth computer running entirely different software, as a failover option.)
Post reply on HN