Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

Post reply on HN
Print view View original

241–250 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#241

Post by detaro »

ajross wrote: post #228
tptacek wrote: post #185

Earlier quoted context omitted.

Not handling corrupted data is kind of the point of cryptographic authentication systems. Informally and generally, the first test of a MAC or a signature of any sort is to see if it fails on arbitrary random single bit flips and shifts. The protocol here seems to have done what it was designed to do. The corrupted shard has simply been removed from service, and would be replaced if there was any need. The ecosystem…

So... Yeti isn't broken then? Seems like the protocol does handle it? Seems like there's some confusion on this point in this thread.

The Yeti2022 log is corrupted due to the random event. This has been correctly detected, and is by design and policy not fixable, since logs are not allowed to rewrite their history (ensuring that they don't is very much the point of CT). That the log broke is annoying but not critical, and the consequences are very much CT working as intended.

You can argue if the software running the log should have verified that it calculated the correct thing before publishing it, but that's not a protocol concern.

Re: Single random bit flip causes error in certificate transparency log

#242

Post by twistedpair »

goodpoint wrote: post #158
aseipp wrote: post #124

Earlier quoted context omitted.

This is just learned helplessness because Intel were stingy as shit for over a decade and wanted to segregate their product lines. Error correction is literally prevalent in every single part of every PHY layer in a modern stack, it is an absolute must, and the lack of error correction in RAM is, without question, a ridiculous gap that should have never been allowed in the first place in any modern machine, especiall…

> By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place. Having worked on a good bunch of critical system, there aren't multiple safeguards in most hardware. E.g. a multiplication error in a core will not be detected by an external device. Or a bit flip when reading cache, or from a storage device. Very often the only real safeguard is to do the whole computation tw…

Failing to account for bit flips and HW failure is too common in web/service coding. Lookup how Google dropped a massive Big Table instance in prod, and traced it back to a cosmic ray bit flip that made a WRITE instruction into a DROP TABLE instruction.

I laugh when I compare my day to day coding to that of an avionics programmer in the aero industry.

Re: Single random bit flip causes error in certificate transparency log

#243

Post by xpasky »

twistedpair wrote: post #242
goodpoint wrote: post #158

Earlier quoted context omitted.

> By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place. Having worked on a good bunch of critical system, there aren't multiple safeguards in most hardware. E.g. a multiplication error in a core will not be detected by an external device. Or a bit flip when reading cache, or from a storage device. Very often the only real safeguard is to do the whole computation tw…

Failing to account for bit flips and HW failure is too common in web/service coding. Lookup how Google dropped a massive Big Table instance in prod, and traced it back to a cosmic ray bit flip that made a WRITE instruction into a DROP TABLE instruction. I laugh when I compare my day to day coding to that of an avionics programmer in the aero industry.

I couldn't find it. Do you have a reference?

Re: Single random bit flip causes error in certificate transparency log

#244

Post by goodpoint »

twistedpair wrote: post #242
goodpoint wrote: post #158

Earlier quoted context omitted.

> By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place. Having worked on a good bunch of critical system, there aren't multiple safeguards in most hardware. E.g. a multiplication error in a core will not be detected by an external device. Or a bit flip when reading cache, or from a storage device. Very often the only real safeguard is to do the whole computation tw…

Failing to account for bit flips and HW failure is too common in web/service coding. Lookup how Google dropped a massive Big Table instance in prod, and traced it back to a cosmic ray bit flip that made a WRITE instruction into a DROP TABLE instruction. I laugh when I compare my day to day coding to that of an avionics programmer in the aero industry.

The "web" coding words falls for things like leftpad, imagine talking about bit flips.

It's sad how immature the software industry can be. It's been around for "only" 60/70 years after all.

Re: Single random bit flip causes error in certificate transparency log

#245

Post by admax88q »

psanford wrote: post #205
admax88q wrote: post #198

Earlier quoted context omitted.

Conceivably you could also fix this by having all verifiets special case this one certificate in their verification software to substitute the correct hash? Obviously that's a huge pain but in theory it would work?

You really want to make everyone special case this because 1 CT log server had a hardware failure? This is not the first time a log server had to be removed due to a failure, nor will it be the last. The whole protocol is designed to be resilient to this. What would be the point of doing something besides following the normal procedures around log failures?

I was just asking to check if I understood the problem correctly.

Re: Single random bit flip causes error in certificate transparency log

#246

Post by kingosticks »

fmajid wrote: post #49
layoutIfNeeded wrote: post #2

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

I’ve had a case where a bit flip in a TCP stream was not caught because it happened in a Singapore government deep packet inspection snoop gateway that recalculated the TCP checksum for the bit-flipped segment: https://blog.majid.info/telco-snooping/

Sorry if this is obvious, but how do we know this isn't due to something more "innocent" like fragmentation?

Re: Single random bit flip causes error in certificate transparency log

#247

Post by theevilsharpie »

willis936 wrote: post #226
theevilsharpie wrote: post #215

Earlier quoted context omitted.

> This is 4 years old now, but does produce some interesting results. > https://hardwarecanucks.com/cpu-motherboard/ecc-memory-amds- ... The author of that article doesn't have hands-on experience with ECC DRAM, and mistakenly concludes that ECC on Ryzen is unreliable because of a misunderstanding of how Linux behaves when it encounters an uncorrected error. However, the author at least includes screenshots which sho…

The AM4 platform does not support ECC reporting. No motherboard can fix this. https://www.servethehome.com/asrock-rack-x570d4u-2l2t-review...

ECC reporting is part of the memory controller (which is unified across all Zen architecture parts), and is fully supported and functional. You can see the reporting working as expected within the Hardware Canucks article linked in the grand parent.

The article you linked mentions that ECC reporting is not working with the on-board IPMI controller (which presumably means that ECC events aren't being logged in the SEL). While that might be a limitation of this board (and other IPMI-equipped AM4 boards), reporting from within the operating system will still work.

Re: Single random bit flip causes error in certificate transparency log

#248

Post by noxer »

tialaramex wrote: post #229
noxer wrote: post #177

Earlier quoted context omitted.

>I haven't thought about whether an actual blockchain is really the best solution... Most likely not. But the tech behind FBA (Federated Byzantine Agreement) distributed ledgers would make an extremely reliable system that can handle malfunction of hardware and large outages of nodes. And since this is a write-only log and only some entities can write to it, it could be implemented with permission so that the system…

Technically everyone can write to it. However you can only write certain specific things. In the case of Yeti 2022 you were only able to log (pre-)certificates signed by particular CAs trusted in the Web PKI, which were due to expire in the year 2022. In practice the vast majority of such logging is done by issuing CAs, as part of their normal operations. But it is possible (and is done purposefully, at least sometim…

Thats pretty much the default state of any blockchain like systems. You need a private key to write to it. Its just that in most public blockchains can have an infinite amount of new private keys can be generated an and some kind of token is attached to it. For a log none of that would be needed. A central operator could hand out keys to anyone who should be able to write to it and for all other its read-only. And ofc the key alone would still not allow someone to write invalid data.

Re: Single random bit flip causes error in certificate transparency log

#249

Post by agilob »

jart wrote: post #65

The push for crypto without ecc ram is a nonstop horror show. Software under normal circumstances is remarkably resilient to having its memory corrupted. However crypto algorithms are designed so that a single bit flip effectively changes all the bits in a block. If you chain blocks then a single bit flip in one block destroys all the blocks. I've seen companies like msps go out of business because they were doing cr…

>companies like msps go out of business because they were doing crypto with consumer hardware

any story on this? can you elaborate or add something to read about it?

Re: Single random bit flip causes error in certificate transparency log

#250

Post by tialaramex »

admax88q wrote: post #245
psanford wrote: post #205

Earlier quoted context omitted.

You really want to make everyone special case this because 1 CT log server had a hardware failure? This is not the first time a log server had to be removed due to a failure, nor will it be the last. The whole protocol is designed to be resilient to this. What would be the point of doing something besides following the normal procedures around log failures?

I was just asking to check if I understood the problem correctly.

Yes, in principle you could special case all the verifiers, although that's an open set, the logs are public.
Post reply on HN