Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

Post reply on HN
Print view View original

81–90 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#81

Post by fennecfoxen »

MayeulC wrote: post #71
joshspankit wrote: post #59

Earlier quoted context omitted.

I had thought that Starlink would become extremely compelling when the servers were in orbit as well, but maybe that’s naive. Cubesats with massive arrays of active storage might be far too difficult (aka costly) to protect properly.

One of the issues with putting servers in orbit is cooling; you can't just use fans in space. On the other hand, real estate is pretty cheap. Servicing is hard, though, and micrometeorites are another risk. Plus launch costs being high, and radiation an issue, I don't see it happening any time soon outside of very specific areas.

Forget cooling for a minute. Cooling dissipates energy but you need to collect the energy first and worry about powering the server. It’s going to be a solar-plus-battery system, which is heavy and expensive, with substantial launch costs.

Re: Single random bit flip causes error in certificate transparency log

#82

Post by RL_Quine »

eru wrote: post #66
formerly_proven wrote: post #43

Earlier quoted context omitted.

According to the Intel developer's manual L1 has parity and all caches up from that have ECC. This would seem to imply that the ring / mesh also has at least parity (to retry on error). Parity instead of ECC on L1(D) makes sense since the L1(D) has to handle small writes well, while the other caches deal in lines.

Of course, parity can only detect an odd number of bit flips.

Some systems do have two parity bits but obviously there’s efficiency loss there.

Re: Single random bit flip causes error in certificate transparency log

#84

Post by XorNot »

gbrown_ wrote: post #10

Can't say I really like the title as it comes across as an absolute statement whereas the bit flip could have happened for any number of unknown reasons. I saw a similar take on Twitter and whilst root causing such things (especially when it's a single occurrence) isn't always possible, shrugging and saying "cosmic rays" should be the last thing to posit not one of the first.

Why? When you do Raman spectroscopy, in any given session you're likely to see 1 going through a 1 cm^2 sample. Students are taught that if you get a super sharp increasing looking peak, first rerun it because you probably just saw a cosmic ray.

Re: Single random bit flip causes error in certificate transparency log

#85

Post by superjan »

Kimitri wrote: post #14
superjan wrote: post #3

Earlier quoted context omitted.

I am used to ‘bit rot’ refer to code becoming obsolete due to lack of maintenance. Can’t we use another term for actual hardware errors?

It may be a regional thing but I have never heard ”bit rot” refer to legacy code. In the retro computing circles bit rot refers to hardware defects (usually floppies or other storage media) caused by cosmic rays or other environmental hazards.

Yeah looks like I was confusing “software rot” and “bit rot”.

https://en.wikipedia.org/wiki/Software_rot

Re: Single random bit flip causes error in certificate transparency log

#86

Post by rob_c »

jacquesm wrote: post #53
dannyw wrote: post #17

Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

Process enough data and even ECC can - and will - fail undetected. Any kind of mechanism you come up with is going to have some rate of undetected errors.

Given the rate required for this its not a reasonable assumption. It's like saying Amazon sees sha256 collisions between S3 buckets. Just doesn't happen in practice.

Re: Single random bit flip causes error in certificate transparency log

#87

Post by willis936 »

fulafel wrote: post #30
dannyw wrote: post #17

Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

These things exist, using trade names like chipkill or lockstep memory. Though they don't need to sacrifice half of the memory chips to get good error recovery properties. Note that this is still not end-to-end protection of data integrity. Bit flips happen in networking, storage, buses between everything, caches, CPUs, etc. See eg [1] [1] https://arxiv.org/abs/2102.11245 Silent Data Corruptions at Scale (based on em…

Checksumming filesystems and file transfer protocols cover many cases. SCP, rsync, btrfs, and zfs all fix this problem.

As for guaranteeing the computed data is correct: I know space systems often have two redundant computers that calculate everything and compare results. It's crazy expensive and power demanding, but it all but solves the problem.

Re: Single random bit flip causes error in certificate transparency log

#88

Post by dathinab »

dannyw wrote: post #17

Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

In my experience it's better to run redundancy on a higher abstraction level.

I.e. (simplified) you do the computation twice on different systems then interchange hashes of the result and if they match you continue.

Re: Single random bit flip causes error in certificate transparency log

#90

Post by juped »

solarkraft wrote: post #42
juped wrote: post #24

Earlier quoted context omitted.

"cosmic rays" are more of a well-known term of art for "single bit flipped with unknown hardware cause" than a reference to literal cosmic rays

Interesting, because I was thinking of ... cosmic rays.

sometimes it really is cosmic rays! but you usually can't know after the fact
Post reply on HN