Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

Post reply on HN
Print view View original

131–140 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#131

Post by quesera »

drumbaby wrote: post #93
juped wrote: post #24

Earlier quoted context omitted.

"cosmic rays" are more of a well-known term of art for "single bit flipped with unknown hardware cause" than a reference to literal cosmic rays

I remember an older less-computer savvy gentlemen asking for support because "the program isn't working", when after a few questions we realized his computer won't boot up (screen dark, etc.). We thought his terminology was all screwed up. But now I realize he just lacked the necessary PR skill. He should have said that "the program isn't working" is a well-known term of art for power users such as himself. The fact…

Go easy on the old guy. It sounds to me like he was completely correct, at the appropriate level of abstraction for him.

And as for cosmic rays...this may be a sensitive topic for you (or maybe you're just in a condescending mood), so I'll tread carefully, but seems simple to me.

The actual cause of the problem, given the instrumentation in place at the time of incident, is unknowable. But it might have been a subatomic particle. It absolutely definitively is, sometimes.

Metonymy might be imprecise, but it's human. I'm not sure what standard you intend to hold commenters here (or old guys with computers) to.

Re: Single random bit flip causes error in certificate transparency log

#132

Post by willis936 »

judge2020 wrote: post #104
cletus wrote: post #67

Cosmic-ray bit flipping is real and it has real security concerns. This also makes Intel's efforts at market segmentation by not having ECC support in any consumer CPUs [1] even more unforgivable and dangerous. Example: bitsquatting on domains [2]. [1]: https://arstechnica.com/gadgets/2021/01/linus-torvalds-blame... [2]: https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...

Also, ECC ram is technically supported on AMD’s recent consumer platform, although it’s not advertised as so since they don’t do validation testing for it.

I've read that reporting of ECC events is not supported on consumer Ryzen. It's not a complete solution and since unregistered ECC is being used, how can you even be sure the memory controller is doing any error correction at all?

Someone would need to induce memory errors and publish their results. I'd love to read it.

This is 4 years old now, but does produce some interesting results.

https://hardwarecanucks.com/cpu-motherboard/ecc-memory-amds-...

Re: Single random bit flip causes error in certificate transparency log

#133

Post by zinekeller »

ajross wrote: post #129
lima wrote: post #120

Earlier quoted context omitted.

> This is a failed cert, which is a failure that can be reasonably easily handled by manual intervention. This isn't a misissued cert that can be revoked, it permanently breaks the CT log in question since the error propagates down the chain.

It doesn't break anything that I can see (though I'm no expert on the particular protocol). Our ability to detect bad certs isn't compromised, precisely because this was noticed by human beings who can adjust the process going forward to work around this. Really the bigger news here seems to be a software bug: the CT protocol wasn't tolerant of bad input data and was trusting actors that clearly can't be trusted full…

Your statement is, to be frank, non-sensical. The protocol itself isn't broken, at least for previous Yeti instances, certificate data are correctly parsed and rejected.* In this instance, it seems that the data is verified already pre-signing BUT was flipped mid-signing. This isn't the fault of how CT was designed but rather a hardware failure that requires correction there. (Or at least that's the likely explanation, it could be a software bug° but it will be a very consistent and obvious behaviour if it is indeed a software bug.)

On the issue of subsequent invalidation of all submitted certificates, this is prevented by submitting to at least 3 different entities (as of now, there's a discussion whether if this should be increased), so if a log is subsequently found to be corrupted, the operator can send a "operator error" signal to the browser, and any tampered logs are blacklisted from browsers. (Note that all operators of CT lists are members of CA/B forum, at least as of 2020. In standardisation phase, some individuals have operated their own servers but this is no longer true.)

* Note that if the cert details are nonsensical but technically valid, it is still accepted by design, because all pre-certificates are countersigned by the intermediate signer (which the CT log operator checks from known roots). If the intermediate is compromised, then the correct response is obviously a revocation and possibly distrust.

° At least the human-induced variety, you could say that this incident is technically a software bug that occurred due to a hardware fault.

Re: Single random bit flip causes error in certificate transparency log

#134

Post by fulafel »

jart wrote: post #65

The push for crypto without ecc ram is a nonstop horror show. Software under normal circumstances is remarkably resilient to having its memory corrupted. However crypto algorithms are designed so that a single bit flip effectively changes all the bits in a block. If you chain blocks then a single bit flip in one block destroys all the blocks. I've seen companies like msps go out of business because they were doing cr…

How did the company go out of business?

Re: Single random bit flip causes error in certificate transparency log

#136

Post by mtkd »

aseipp wrote: post #124
ajross wrote: post #110

Earlier quoted context omitted.

> The push for crypto without ecc ram is a nonstop horror show That's a bit hyperbolic. First, ECC doesn't protect the full data chain, you can have a bitflip in a hardware flip flop (or latch open a gate that drains a line, etc...) before the value reaches the memory. Logic is known to glitch too. Second: ECC is mostly designed to protect long term storage in DRAM. Recognize that a cert like this is a very short-ter…

This is just learned helplessness because Intel were stingy as shit for over a decade and wanted to segregate their product lines. Error correction is literally prevalent in every single part of every PHY layer in a modern stack, it is an absolute must, and the lack of error correction in RAM is, without question, a ridiculous gap that should have never been allowed in the first place in any modern machine, especiall…

'stingy as shit' or maximising short-term shareholder value -- the hardware is possibly not the only broken model here

Re: Single random bit flip causes error in certificate transparency log

#137

Post by ineedasername »

ajross wrote: post #110
jart wrote: post #65

The push for crypto without ecc ram is a nonstop horror show. Software under normal circumstances is remarkably resilient to having its memory corrupted. However crypto algorithms are designed so that a single bit flip effectively changes all the bits in a block. If you chain blocks then a single bit flip in one block destroys all the blocks. I've seen companies like msps go out of business because they were doing cr…

> The push for crypto without ecc ram is a nonstop horror show That's a bit hyperbolic. First, ECC doesn't protect the full data chain, you can have a bitflip in a hardware flip flop (or latch open a gate that drains a line, etc...) before the value reaches the memory. Logic is known to glitch too. Second: ECC is mostly designed to protect long term storage in DRAM. Recognize that a cert like this is a very short-ter…

That's a bit hyperbolic.

Would ECC have avoided the issue in this case? If so then it's hard not to agree that is should be considered a minimum standard. It looks like Yeti 2022 isn't going to survive this intact, and while they can resolve the issues in other ways, not everyone will always be so fortunate and ECC is a relatively small step to avoid a larger problem.

Re: Single random bit flip causes error in certificate transparency log

#138

Post by gbrown_ »

wolf550e wrote: post #121
gbrown_ wrote: post #10

Can't say I really like the title as it comes across as an absolute statement whereas the bit flip could have happened for any number of unknown reasons. I saw a similar take on Twitter and whilst root causing such things (especially when it's a single occurrence) isn't always possible, shrugging and saying "cosmic rays" should be the last thing to posit not one of the first.

Hardware issues that cause bitflips like that one famous issue[1] with Sun servers in ~2000 (supposedly caused by radioactive isotopes in IBM manufactures SRAM chips) are often called "cosmic rays". 1 - https://www.computerworld.com/article/2585216/mcnealy-blames...

> are often called "cosmic rays".

No! This exact issue is a classic example because at one point the fault was attributed to cosmic rays by Sun before the true cause came to light. As another commenter has said, terminology matters.

Re: Single random bit flip causes error in certificate transparency log

#140

Post by dijit »

ZiiS wrote: post #68
dijit wrote: post #21

Earlier quoted context omitted.

These are all very fair statements but there’s no guarantee that ECC memory was even used. Computers typically fail open when ECC is potentially present but not available. People also cite early stage google and intentionally do not buy ECC components, running more consumer hardware for production workloads. Even if google later recanted that theology.

Public CAs are not that type of people; I would be disapointed if that were not running two seperate systems checking each other for consistancy; having top of the range ECC running well inside its specification must be table stakes.

> Public CAs are not that type of people

I think you hold public CAs to a higher standard than many hold themselves to.

There are hundreds of CAs and many (if not most) are shockingly awful.

Which is why we have had a huge push back against the PKI cartels.

Post reply on HN