Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

Post reply on HN
Print view View original

101–110 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#101

Post by willis936 »

MayeulC wrote: post #71
joshspankit wrote: post #59

Earlier quoted context omitted.

I had thought that Starlink would become extremely compelling when the servers were in orbit as well, but maybe that’s naive. Cubesats with massive arrays of active storage might be far too difficult (aka costly) to protect properly.

One of the issues with putting servers in orbit is cooling; you can't just use fans in space. On the other hand, real estate is pretty cheap. Servicing is hard, though, and micrometeorites are another risk. Plus launch costs being high, and radiation an issue, I don't see it happening any time soon outside of very specific areas.

Radiation cooling is pretty dramatic in space. With an isolated radiation shield/mirror from the sun and a decent sized heatsink, you can have a lot of cooling power on hand. A server isn't going to be pumping out nearly as much heat as a rocket engine, and launch vehicles don't tend to overheat.

Re: Single random bit flip causes error in certificate transparency log

#102

Post by tialaramex »

pdpi wrote: post #36
blumomo wrote: post #26

could also be an excuse to cover the real circumstances

It could, yes. I’m sure the thought also crossed the mind of everybody else in that thread, and they’ve clearly dismissed it. Bare conspiratorial assertions don’t advance the conversation in any meaningful way. If you have something constructive to add to the discussion, by all means do so. Otherwise, please refrain from this sort of comment

Humans can examine the underlying data and reason that indeed one bitflip could cause the observed result.

In this particular case the alternatives are:

1. The bitflip happened. Cosmic rays. Defective CPU. Maybe a tremendously rare data tearing race somewhere in software.

OR

2a. Somebody has a fast way to generate near-Collisions in SHA256, which are 1-bit errors from the target hash. Rather than publish this breakthrough they:

2b. Had a public CA issue and sign two certificates, one uninteresting real certificate for *.molabtausnipnimo.ml which we've now seen and another "evil" near-colliding certificate with the one-bit-different hash then they:

2c. Logged the real certificate but at only one log (Yeti) they get the log to show the real certificate yet actually use the hash from the "evil" certificate, thus perhaps obtaining a single valid SCT for the "evil" certificate.

Scenario 1 isn't terribly likely for any one certificate, but given we're logging vast numbers every day and errors inherently cascade in this system, it was going to happen sooner or later.

Scenario 2 fails for having too many conspirators (a public CA, and likely a different public log) and yet it achieves almost nothing even if it remains secret. Your Chrome or Safari expects at least two SCTs (one of them from Google, in this case Google's Argon 2022 log), one doesn't get the job done. And it was never going to remain secret, this occurrence was spotted after less than a day, and thoroughly investigated, resulting in the log being shut down, after just a few days.

[Edited multiple times to fix awful HN formatting.]

Re: Single random bit flip causes error in certificate transparency log

#103

Post by noduerme »

drumbaby wrote: post #93
juped wrote: post #24

Earlier quoted context omitted.

"cosmic rays" are more of a well-known term of art for "single bit flipped with unknown hardware cause" than a reference to literal cosmic rays

I remember an older less-computer savvy gentlemen asking for support because "the program isn't working", when after a few questions we realized his computer won't boot up (screen dark, etc.). We thought his terminology was all screwed up. But now I realize he just lacked the necessary PR skill. He should have said that "the program isn't working" is a well-known term of art for power users such as himself. The fact…

You're saying you have to engineer chip components so they're small enough not to be hit as often by cosmic rays, but large enough to avoid quantum fluctuations? Are quantum fluctuations something Intel is dealing with regularly as they get down under 3nm?

Re: Single random bit flip causes error in certificate transparency log

#104

Post by judge2020 »

cletus wrote: post #67

Cosmic-ray bit flipping is real and it has real security concerns. This also makes Intel's efforts at market segmentation by not having ECC support in any consumer CPUs [1] even more unforgivable and dangerous. Example: bitsquatting on domains [2]. [1]: https://arstechnica.com/gadgets/2021/01/linus-torvalds-blame... [2]: https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...

Also, ECC ram is technically supported on AMD’s recent consumer platform, although it’s not advertised as so since they don’t do validation testing for it.

Re: Single random bit flip causes error in certificate transparency log

#105

Post by jeffbee »

rob_c wrote: post #86
jacquesm wrote: post #53

Earlier quoted context omitted.

Process enough data and even ECC can - and will - fail undetected. Any kind of mechanism you come up with is going to have some rate of undetected errors.

Given the rate required for this its not a reasonable assumption. It's like saying Amazon sees sha256 collisions between S3 buckets. Just doesn't happen in practice.

Undetected ECC errors are common enough to see from time to time in the wild. This paper estimates that a supercomputer sees one undetected error per day.

https://www.cs.virginia.edu/~gurumurthi/papers/asplos15.pdf

Re: Single random bit flip causes error in certificate transparency log

#106

Post by my123 »

judge2020 wrote: post #104
cletus wrote: post #67

Cosmic-ray bit flipping is real and it has real security concerns. This also makes Intel's efforts at market segmentation by not having ECC support in any consumer CPUs [1] even more unforgivable and dangerous. Example: bitsquatting on domains [2]. [1]: https://arstechnica.com/gadgets/2021/01/linus-torvalds-blame... [2]: https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...

Also, ECC ram is technically supported on AMD’s recent consumer platform, although it’s not advertised as so since they don’t do validation testing for it.

Not on all of it. They only don't lock it down on CPUs, but not on APUs.

On AMD SoCs with integrated GPUs, ECC is not usable except if you buy the Ryzen Pro variant. And those are all laptops and most desktops...

Re: Single random bit flip causes error in certificate transparency log

#107

Post by jeffbee »

willis936 wrote: post #91
Jolter wrote: post #40

Earlier quoted context omitted.

IIRC, EEC memory can correct for single-but flips. It can detect/warn/fail on double-bit flips. And it can not detect triple-bit flips. This might be a simplified understanding, but if this has happened only once, that seems to match up with my intuitive understanding of the probability of a triple bit flip occurring in a particular system.

You multiply the probability of two random events to get the probability they will happen at the same time. If the expected value of a bit flip is 10^-18 then two would be 10^-36 and three would he 10^-54. At some point it becomes a philosophical question of how much can the tails of the distribution be tolerated. We've never seen a quantum fluctuation make a whale appear in the sky.

DRAM failures are not independent events, so it’s not appropriate to multiply the probabilities like that. Faults are often clustered in a row, column, bank, page or whatever structure your DRAM has, raising the probability of multi-bit errors.

Re: Single random bit flip causes error in certificate transparency log

#109

Post by tgv »

Waterluvian wrote: post #89

Are “cosmic rays” actually the only or primary way bits get flipped? Or is it just a stand-in for “all the ways non-ECC RAM can spontaneously have an erroneous bit or two”?

Well, "row hammer" notoriously can do that too.

Re: Single random bit flip causes error in certificate transparency log

#110

Post by ajross »

jart wrote: post #65

The push for crypto without ecc ram is a nonstop horror show. Software under normal circumstances is remarkably resilient to having its memory corrupted. However crypto algorithms are designed so that a single bit flip effectively changes all the bits in a block. If you chain blocks then a single bit flip in one block destroys all the blocks. I've seen companies like msps go out of business because they were doing cr…

> The push for crypto without ecc ram is a nonstop horror show

That's a bit hyperbolic.

First, ECC doesn't protect the full data chain, you can have a bitflip in a hardware flip flop (or latch open a gate that drains a line, etc...) before the value reaches the memory. Logic is known to glitch too.

Second: ECC is mostly designed to protect long term storage in DRAM. Recognize that a cert like this is a very short-term value, it's computed and then transmitted. The failure happened fast, before copies of the correct value were made. That again argues to a failure location other than a DRAM cell.

But mostly... this isn't the end of the world. This is a failed cert, which is a failure that can be reasonably easily handled by manual intervention. There have been many other mistaken certs distributed that had to be dealt with manually: they can be the result of software bugs, they can be generated with the wrong keys, the dates can be set wrong, they can be maliciously issued, etc... The system is designed to include manual validation in the loop and it works.

So is ECC a good idea? Of course. Does it magically fix problems like this? No. Is this really a "nonstop horror show"? Not really, we're doing OK.

Post reply on HN