Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

Post reply on HN
Print view View original

211–220 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#211

Post by caf »

zinekeller wrote: post #133
ajross wrote: post #129

Earlier quoted context omitted.

It doesn't break anything that I can see (though I'm no expert on the particular protocol). Our ability to detect bad certs isn't compromised, precisely because this was noticed by human beings who can adjust the process going forward to work around this. Really the bigger news here seems to be a software bug: the CT protocol wasn't tolerant of bad input data and was trusting actors that clearly can't be trusted full…

Your statement is, to be frank, non-sensical. The protocol itself isn't broken, at least for previous Yeti instances, certificate data are correctly parsed and rejected.* In this instance, it seems that the data is verified already pre-signing BUT was flipped mid-signing. This isn't the fault of how CT was designed but rather a hardware failure that requires correction there. (Or at least that's the likely explanatio…

Presumably it's possible to code defensively against this sort of thing, by eg. running the entire operation twice and checking the result is the same before committing it to the published log?

Re: Single random bit flip causes error in certificate transparency log

#212

Post by rkangel »

cletus wrote: post #67

Cosmic-ray bit flipping is real and it has real security concerns. This also makes Intel's efforts at market segmentation by not having ECC support in any consumer CPUs [1] even more unforgivable and dangerous. Example: bitsquatting on domains [2]. [1]: https://arstechnica.com/gadgets/2021/01/linus-torvalds-blame... [2]: https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...

I don't understand the issue with market segmentation here. I can absolutely see the reason why all of my servers should have ECC, but I don't see why my gaming PC should (or even my work development machine). What's the worst case impact of the (extremely rare) bit-flip on one of those machines?

Re: Single random bit flip causes error in certificate transparency log

#213

Post by makomk »

caf wrote: post #211
zinekeller wrote: post #133

Earlier quoted context omitted.

Your statement is, to be frank, non-sensical. The protocol itself isn't broken, at least for previous Yeti instances, certificate data are correctly parsed and rejected.* In this instance, it seems that the data is verified already pre-signing BUT was flipped mid-signing. This isn't the fault of how CT was designed but rather a hardware failure that requires correction there. (Or at least that's the likely explanatio…

Presumably it's possible to code defensively against this sort of thing, by eg. running the entire operation twice and checking the result is the same before committing it to the published log?

Big tech companies like Google and Facebook have encountered problems where running the same crypto operation twice on the same processor deterministically or semi-deterministically gives the same incorrect result... so the check needs to be on done on separate hardware as well.

Re: Single random bit flip causes error in certificate transparency log

#214

Post by DevKoala »

Shikadi wrote: post #207
DevKoala wrote: post #206

Earlier quoted context omitted.

I was going to comment, but you edited your post. Yes, it is worse than we usually think on the software side.

I still think it made sense up until now to not bother with it on consumer hardware, and even at this point. The probability of your phone having a software glitch needing a reboot is way higher. Now that it's practically a free upgrade? Should be included by default. But I still don't think it's nearly as nefarious as people make it out to be that it has been this way for so long

I don’t think they are an issue in a phone, but in a system like a blockchain that goes through so much effort to achieve consistency, the severity of the error is magnified hence the lesser tolerance for the error rate.

Re: Single random bit flip causes error in certificate transparency log

#215

Post by theevilsharpie »

willis936 wrote: post #132
judge2020 wrote: post #104

Earlier quoted context omitted.

Also, ECC ram is technically supported on AMD’s recent consumer platform, although it’s not advertised as so since they don’t do validation testing for it.

I've read that reporting of ECC events is not supported on consumer Ryzen. It's not a complete solution and since unregistered ECC is being used, how can you even be sure the memory controller is doing any error correction at all? Someone would need to induce memory errors and publish their results. I'd love to read it. This is 4 years old now, but does produce some interesting results. https://hardwarecanucks.com/cp…

> This is 4 years old now, but does produce some interesting results.

> https://hardwarecanucks.com/cpu-motherboard/ecc-memory-amds-...

The author of that article doesn't have hands-on experience with ECC DRAM, and mistakenly concludes that ECC on Ryzen is unreliable because of a misunderstanding of how Linux behaves when it encounters an uncorrected error. However, the author at least includes screenshots which show ECC functionality on Ryzen working properly.

> ...since unregistered ECC is being used, how can you even be sure the memory controller is doing any error correction at all?

ECC is performed by the memory controller, and requires an extra memory device per rank and 8 extra data bits, which unbuffered ECC DIMMs provide.

Registered memory has nothing to do with ECC (although in practice, registered DIMMs almost always have ECC support). It's simply a mechanism to reduce electrical load on the memory controller to allow for the usage of higher-capacity DIMMs than what unbuffered DIMMs would allow.

With respect to Ryzen, Zen's memory controller architecture is unified, and owners of Ryzen CPUs use the same memory controller found in similar-generation Threadripper and EPYC processors (just fewer of them). Although full ECC support is not required on the AM4 platform specifically (it's an optional feature that can be implemented by the motherboard maker), it's functional and supported if present. Indeed, there are several Ryzen motherboards aimed at professional audiences where ECC is an explicitly advertised feature of the board.

Re: Single random bit flip causes error in certificate transparency log

#216

Post by kevin_thibedeau »

Shikadi wrote: post #203
aseipp wrote: post #124

Earlier quoted context omitted.

This is just learned helplessness because Intel were stingy as shit for over a decade and wanted to segregate their product lines. Error correction is literally prevalent in every single part of every PHY layer in a modern stack, it is an absolute must, and the lack of error correction in RAM is, without question, a ridiculous gap that should have never been allowed in the first place in any modern machine, especiall…

People who tout this don't understand the probability of bit flips. It's measured in failures per _billion_ hours of operation. This matters a ton in an environment with thousands of memory modules (data centers and super computers) but you're lucky to experience a single ram bit flip more than once or twice in your entire life Edit: there's some new (to me) information from real world results, interesting read. http…

Bit flips are guaranteed to happen in digital systems. No matter how low the probability is, it will never be zero. You can't go around thinking you're going to dodge a bullet because its unlikely. If it weren't for the pervasive use of error detection in common I/O protocols you would be subjected to these errors much more frequently.

Re: Single random bit flip causes error in certificate transparency log

#217

Post by X6S1x6Okd1st »

bonzini wrote: post #160
belter wrote: post #112

Earlier quoted context omitted.

If they have two computers and results differ, how do they decide which one is correct ? ;-)

Usually they have an odd number. The Space Shuttle had five, of which the fifth was running completely different software. In case of a 2/2 split the crew could shut down a pair (in case the failure was clear) or switch to the backup computer.

Interestingly etherum also takes the completely different software approach as well: there are 5 different main clients

Re: Single random bit flip causes error in certificate transparency log

#218

Post by sgtnoodle »

jeffbee wrote: post #209
BenjiWiebe wrote: post #208

Earlier quoted context omitted.

I believe the usual concern is bit flips due to subatomic particles, and as far as I'm aware that only flips one bit per particle.

I don't see why a high-energy particle strike would confine itself to a single bit. The paper I posted elsewhere in this thread says that "the most likely cause of the higher single-bit, single-column, and single-bank transient fault rates in Cielo is particle strikes from high-energy neutrons". In the paper, both single-bit and multi-bit errors are sensitive to altitude.

A single particle strike would only affect a single transistor. If that transistor controls a whole column of memory, then sure it could corrupt lots of bits. With ECC, though, it would probably result in a bunch of ECC blocks with a single bit flip, rather than a single ECC block with several bit flips.

Re: Single random bit flip causes error in certificate transparency log

#219

Post by sillysaurusx »

agwa wrote: post #183

OP here. Unless you work for a certificate authority or a web browser, this event will have zero impact on you. While this particular CT log has failed, there are many other CT logs, and certificates are required to be logged to 2-3 different logs (depending on certificate lifetime) so that if a log fails web browsers can rely on one of the other logs to ensure the certificate is publicly logged. This is the 8th log…

Is it possible to give an example other than “cosmic ray”? I know it’s an informal short hand, but it also raises the question of what’s actually causing these flips.

Is it just random stray electrons that happen to stray too far while traveling through the RAM? Very interesting to me.

Re: Single random bit flip causes error in certificate transparency log

#220

Post by caf »

jacquesm wrote: post #50
tsegratis wrote: post #45

Do nuclear missiles have this anywhere? bool launch = false; if(launch) ...

It doesn't really matter whether they have if (launch1 && launch2 && launch3) { launch(); } either because a single bit flip (of the code) could still cause a launch. You'd hope that it was at least stored in ROM and that ECC ensures that even if such a bit flips it does not lead straight to Armageddon. There are some 'near miss' stories where a single switch made all the difference: https://www.theatlantic.com/techn…

It should in-principle be possible to write a branch where the code itself is single-bit-error resistant, in pseudo-machine-code something like:

  LOAD [launch1]
  COMPARE 0x89abcdef
  JUMP_IF_NOT_EQUAL [fail_label]
  LOAD [launch2]
  COMPARE 0x01234567
  JUMP_IF_NOT_EQUAL [fail_label]
  LOAD [launch3]
  COMPARE 0xfedcba98
  JUMP_IF_NOT_EQUAL [fail_label]
  BRANCH [launch_label]
You also need to ensure that launch_label and the location of the branch instruction are both more than one bit away from fail_label. You can duplicate the JUMP_IF_NOT_EQUAL instructions as needed - or indeed the whole block before the BRANCH - as necessary to ensure that.
Post reply on HN