Single random bit flip causes error in certificate transparency log • Hacker News Board

Re: Single random bit flip causes error in certificate transparency log

#201

Post by morelikeborelax » Sun, Jul 04, 2021, 7:39 PM UTC

jart wrote: post #65

The push for crypto without ecc ram is a nonstop horror show. Software under normal circumstances is remarkably resilient to having its memory corrupted. However crypto algorithms are designed so that a single bit flip effectively changes all the bits in a block. If you chain blocks then a single bit flip in one block destroys all the blocks. I've seen companies like msps go out of business because they were doing cr…

I used to sell workstations and servers years ago and trying to convince people they needed ECC Ram and that it was just insurance for (often) just the price of the extra chips on the DIMMs was a nightmare.

The amount of uninformed and inexperienced counter arguments online suggesting it was purely Intel seeking extra money (even though they didn't sell RAM) was ridiculous.

I never understood why there was so much push back from the consumer world commenting on something they had no idea about. Similar arguments for why would you ever need xxGB of RAM while also condemning the incorrect 640kb RAM Bill Gates comment.

Re: Single random bit flip causes error in certificate transparency log

#202

Post by dundarious » Sun, Jul 04, 2021, 7:55 PM UTC

tialaramex wrote: post #186

dundarious wrote: post #154

Earlier quoted context omitted.

I haven't thought about whether an actual blockchain is really the best solution, but the redundancy argument is legitimate. We've been doing it for decades in other systems where an unnoticed bit flip results in complete mission failure, such as an Apollo mission crash. I'm not really sure what Yeti 2022 is exactly, so take this with heaps of salt, but it seems like this is a "mission failure" event -- it can no lon…

> I'm not really sure what Yeti 2022 is exactly Sometimes there are problems with certificates in the Web PKI (approximately the certificates your web browser trusts to determine that this is really news.ycombinator.com, for example). It's a lot easier to discover such problems early, and detect if they've really stopped happening after someone says "We fixed it" if you have a complete list of every certificate. The…

Thank you, that's an excellent description. The CT system as a whole does appear to have ample redundancy, with automated tools informing manual intervention that resolves this individual failure.

Re: Single random bit flip causes error in certificate transparency log

#203

Post by Shikadi » Sun, Jul 04, 2021, 8:30 PM UTC

aseipp wrote: post #124

ajross wrote: post #110

Earlier quoted context omitted.

> The push for crypto without ecc ram is a nonstop horror show That's a bit hyperbolic. First, ECC doesn't protect the full data chain, you can have a bitflip in a hardware flip flop (or latch open a gate that drains a line, etc...) before the value reaches the memory. Logic is known to glitch too. Second: ECC is mostly designed to protect long term storage in DRAM. Recognize that a cert like this is a very short-ter…

This is just learned helplessness because Intel were stingy as shit for over a decade and wanted to segregate their product lines. Error correction is literally prevalent in every single part of every PHY layer in a modern stack, it is an absolute must, and the lack of error correction in RAM is, without question, a ridiculous gap that should have never been allowed in the first place in any modern machine, especiall…

People who tout this don't understand the probability of bit flips. It's measured in failures per _billion_ hours of operation. This matters a ton in an environment with thousands of memory modules (data centers and super computers) but you're lucky to experience a single ram bit flip more than once or twice in your entire life

Edit: there's some new (to me) information from real world results, interesting read. https://www.zdnet.com/article/dram-error-rates-nightmare-on-...

Looks like things are worse than I thought (but still better than most people seem to think). Interesting to note that the motherboard used affects error rate, and it seems that part of it is a luck of the draw situation where some dimms have more errors than others despite being the same manufacturer

Re: Single random bit flip causes error in certificate transparency log

#204

Post by teddyh » Sun, Jul 04, 2021, 8:35 PM UTC

dannyw wrote: post #17

Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

User ‘JoshTriplett’ here suggests using forward error correction (FEC) in RAM:

https://news.ycombinator.com/item?id=11604918

Re: Single random bit flip causes error in certificate transparency log

#205

Post by psanford » Sun, Jul 04, 2021, 8:40 PM UTC

admax88q wrote: post #198

tialaramex wrote: post #186

Earlier quoted context omitted.

> I'm not really sure what Yeti 2022 is exactly Sometimes there are problems with certificates in the Web PKI (approximately the certificates your web browser trusts to determine that this is really news.ycombinator.com, for example). It's a lot easier to discover such problems early, and detect if they've really stopped happening after someone says "We fixed it" if you have a complete list of every certificate. The…

Conceivably you could also fix this by having all verifiets special case this one certificate in their verification software to substitute the correct hash? Obviously that's a huge pain but in theory it would work?

You really want to make everyone special case this because 1 CT log server had a hardware failure?

This is not the first time a log server had to be removed due to a failure, nor will it be the last. The whole protocol is designed to be resilient to this.

What would be the point of doing something besides following the normal procedures around log failures?

Re: Single random bit flip causes error in certificate transparency log

#206

Post by DevKoala » Sun, Jul 04, 2021, 8:46 PM UTC

Shikadi wrote: post #203

aseipp wrote: post #124

Earlier quoted context omitted.

This is just learned helplessness because Intel were stingy as shit for over a decade and wanted to segregate their product lines. Error correction is literally prevalent in every single part of every PHY layer in a modern stack, it is an absolute must, and the lack of error correction in RAM is, without question, a ridiculous gap that should have never been allowed in the first place in any modern machine, especiall…

People who tout this don't understand the probability of bit flips. It's measured in failures per _billion_ hours of operation. This matters a ton in an environment with thousands of memory modules (data centers and super computers) but you're lucky to experience a single ram bit flip more than once or twice in your entire life Edit: there's some new (to me) information from real world results, interesting read. http…

I was going to comment, but you edited your post. Yes, it is worse than we usually think on the software side.

Re: Single random bit flip causes error in certificate transparency log

#207

Post by Shikadi » Sun, Jul 04, 2021, 9:01 PM UTC

DevKoala wrote: post #206

Shikadi wrote: post #203

Earlier quoted context omitted.

People who tout this don't understand the probability of bit flips. It's measured in failures per _billion_ hours of operation. This matters a ton in an environment with thousands of memory modules (data centers and super computers) but you're lucky to experience a single ram bit flip more than once or twice in your entire life Edit: there's some new (to me) information from real world results, interesting read. http…

I was going to comment, but you edited your post. Yes, it is worse than we usually think on the software side.

I still think it made sense up until now to not bother with it on consumer hardware, and even at this point. The probability of your phone having a software glitch needing a reboot is way higher. Now that it's practically a free upgrade? Should be included by default. But I still don't think it's nearly as nefarious as people make it out to be that it has been this way for so long

Re: Single random bit flip causes error in certificate transparency log

#208

Post by BenjiWiebe » Sun, Jul 04, 2021, 9:18 PM UTC

jeffbee wrote: post #107

willis936 wrote: post #91

Earlier quoted context omitted.

You multiply the probability of two random events to get the probability they will happen at the same time. If the expected value of a bit flip is 10^-18 then two would be 10^-36 and three would he 10^-54. At some point it becomes a philosophical question of how much can the tails of the distribution be tolerated. We've never seen a quantum fluctuation make a whale appear in the sky.

DRAM failures are not independent events, so it’s not appropriate to multiply the probabilities like that. Faults are often clustered in a row, column, bank, page or whatever structure your DRAM has, raising the probability of multi-bit errors.

I believe the usual concern is bit flips due to subatomic particles, and as far as I'm aware that only flips one bit per particle.

Re: Single random bit flip causes error in certificate transparency log

#209

Post by jeffbee » Sun, Jul 04, 2021, 9:35 PM UTC

BenjiWiebe wrote: post #208

jeffbee wrote: post #107

Earlier quoted context omitted.

DRAM failures are not independent events, so it’s not appropriate to multiply the probabilities like that. Faults are often clustered in a row, column, bank, page or whatever structure your DRAM has, raising the probability of multi-bit errors.

I believe the usual concern is bit flips due to subatomic particles, and as far as I'm aware that only flips one bit per particle.

I don't see why a high-energy particle strike would confine itself to a single bit. The paper I posted elsewhere in this thread says that "the most likely cause of the higher single-bit, single-column, and single-bank transient fault rates in Cielo is particle strikes from high-energy neutrons". In the paper, both single-bit and multi-bit errors are sensitive to altitude.

Re: Single random bit flip causes error in certificate transparency log

#210

Post by joshspankit » Sun, Jul 04, 2021, 9:49 PM UTC

noduerme wrote: post #95

joshspankit wrote: post #59

Earlier quoted context omitted.

I had thought that Starlink would become extremely compelling when the servers were in orbit as well, but maybe that’s naive. Cubesats with massive arrays of active storage might be far too difficult (aka costly) to protect properly.

Putting servers in orbit is a really, really bad idea. Firstly, it would be wildly inefficient, just due to the unavoidable delay both ways. You expect delay over a long distance network, but you want the server to be positioned and cabled up to minimize latency. Just locating a satellite requires quite a bit of overhead, so treating them as servers rather than clients would create a huge amount of latency. As a fail…

> Firstly, it would be wildly inefficient, just due to the unavoidable delay both ways

The point I was thinking about is a scenario where Starlink satellites communicate with each other via laser (already starting to happen), and then communicate with the end user via the satellite over them. Because we're talking about speed-of-light transmission between sats, data in the Starlink network can theoretically cover "ground" (aka miles/km) faster than ground-based ISPs.

Then it makes sense to deploy servers within that network so that two Starlink users can have extremely low latency from user to server to other user (the slowest part being the earth to sat latencies, one for each user).