Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

Post reply on HN
Print view View original

231–240 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#231

Post by astrange »

ericbarrett wrote: post #182
jart wrote: post #65

The push for crypto without ecc ram is a nonstop horror show. Software under normal circumstances is remarkably resilient to having its memory corrupted. However crypto algorithms are designed so that a single bit flip effectively changes all the bits in a block. If you chain blocks then a single bit flip in one block destroys all the blocks. I've seen companies like msps go out of business because they were doing cr…

In my younger days I worked in customer support for a storage company. A customer had one of our products crash (kernel panic) at a sensitive time. We had a couple of crack engineers I used to hover around to try to pick up gdb tricks so I followed this case with interest—it was a new and unexpected panic. Turns out the crash was caused by the processor taking the wrong branch . Something like this (it wasn’t Intel b…

I saw a kernel panic because of this _yesterday_. (Crash where the failing instruction in memory was 1 bit different from on disk.)

Re: Single random bit flip causes error in certificate transparency log

#233

Post by DominoTree »

sillysaurusx wrote: post #219
agwa wrote: post #183

OP here. Unless you work for a certificate authority or a web browser, this event will have zero impact on you. While this particular CT log has failed, there are many other CT logs, and certificates are required to be logged to 2-3 different logs (depending on certificate lifetime) so that if a log fails web browsers can rely on one of the other logs to ensure the certificate is publicly logged. This is the 8th log…

Is it possible to give an example other than “cosmic ray”? I know it’s an informal short hand, but it also raises the question of what’s actually causing these flips. Is it just random stray electrons that happen to stray too far while traveling through the RAM? Very interesting to me.

Old DRAM/SRAM simply becoming flaky, overheating, physical connection issues, faulty power - there are a ton of reasons these flips happen, and I would think that in most cases they are entirely unnoticed or simply observed as an application crashing, rather than being recognized as directly caused by bit flips.

Re: Single random bit flip causes error in certificate transparency log

#234

Post by ericbarrett »

astrange wrote: post #231
ericbarrett wrote: post #182

Earlier quoted context omitted.

In my younger days I worked in customer support for a storage company. A customer had one of our products crash (kernel panic) at a sensitive time. We had a couple of crack engineers I used to hover around to try to pick up gdb tricks so I followed this case with interest—it was a new and unexpected panic. Turns out the crash was caused by the processor taking the wrong branch . Something like this (it wasn’t Intel b…

I saw a kernel panic because of this _yesterday_. (Crash where the failing instruction in memory was 1 bit different from on disk.)

Sounds like a regular RAM bitflip, doesn't it?

Re: Single random bit flip causes error in certificate transparency log

#235

Post by astrange »

londons_explore wrote: post #196

The design of bitcoin prevents such an error, since all other nodes in the network verify everything. I wonder if the CT log should have been more like that...?

All the CT readers do verify it and reject the log, but it looks like the writer trusts itself. So it keeps rebroadcasting a broken log instead of instantly failing. Having an internal blockchain or just another verification server would solve it, I guess.

Re: Single random bit flip causes error in certificate transparency log

#236

Post by tptacek »

ajross wrote: post #228
tptacek wrote: post #185

Earlier quoted context omitted.

Not handling corrupted data is kind of the point of cryptographic authentication systems. Informally and generally, the first test of a MAC or a signature of any sort is to see if it fails on arbitrary random single bit flips and shifts. The protocol here seems to have done what it was designed to do. The corrupted shard has simply been removed from service, and would be replaced if there was any need. The ecosystem…

So... Yeti isn't broken then? Seems like the protocol does handle it? Seems like there's some confusion on this point in this thread.

I think Yeti2022 is just the name for one instance of the global CT log? Nick Lamb could probably say more about this; I understand CT mostly in the abstract, and as the total feed of CT records you surveil for stuff.

Re: Single random bit flip causes error in certificate transparency log

#237

Post by jacquesm »

caf wrote: post #220
jacquesm wrote: post #50

Earlier quoted context omitted.

It doesn't really matter whether they have if (launch1 && launch2 && launch3) { launch(); } either because a single bit flip (of the code) could still cause a launch. You'd hope that it was at least stored in ROM and that ECC ensures that even if such a bit flips it does not lead straight to Armageddon. There are some 'near miss' stories where a single switch made all the difference: https://www.theatlantic.com/techn…

It should in-principle be possible to write a branch where the code itself is single-bit-error resistant, in pseudo-machine-code something like: LOAD [launch1] COMPARE 0x89abcdef JUMP_IF_NOT_EQUAL [fail_label] LOAD [launch2] COMPARE 0x01234567 JUMP_IF_NOT_EQUAL [fail_label] LOAD [launch3] COMPARE 0xfedcba98 JUMP_IF_NOT_EQUAL [fail_label] BRANCH [launch_label] You also need to ensure that launch_label and the location…

Your comment is one of the reasons why I still believe assembly has its place, when it really matters what kind of instructions you put out this is the sort of control you want. What your high level language is going to dump in the instruction stream is totally invisible (and interpreted languages are not even up for consideration in situations like these).

Re: Single random bit flip causes error in certificate transparency log

#238

Post by brokenmachine »

fmajid wrote: post #49
layoutIfNeeded wrote: post #2

Bit rot is real. Last month we’ve had weird linker errors on one of our build servers. Turns out that one of the binary libs in the build cache got a single bit flipped in the symbol table, which changed the symbol name, causing the linker errors. If the bit-flip had occured in the .TEXT section, then it wouldn’t have caused any errors at build time, and we would have released a buggy binary. It might have just crash…

I’ve had a case where a bit flip in a TCP stream was not caught because it happened in a Singapore government deep packet inspection snoop gateway that recalculated the TCP checksum for the bit-flipped segment: https://blog.majid.info/telco-snooping/

Nice article, enjoyed it.

Re: Single random bit flip causes error in certificate transparency log

#239

Post by astrange »

ericbarrett wrote: post #234
astrange wrote: post #231

Earlier quoted context omitted.

I saw a kernel panic because of this _yesterday_. (Crash where the failing instruction in memory was 1 bit different from on disk.)

Sounds like a regular RAM bitflip, doesn't it?

Sure, but even with those you'd usually expect it to happen to data, which is a lot larger.

Re: Single random bit flip causes error in certificate transparency log

#240

Post by detaro »

tptacek wrote: post #236
ajross wrote: post #228

Earlier quoted context omitted.

So... Yeti isn't broken then? Seems like the protocol does handle it? Seems like there's some confusion on this point in this thread.

I think Yeti2022 is just the name for one instance of the global CT log? Nick Lamb could probably say more about this; I understand CT mostly in the abstract, and as the total feed of CT records you surveil for stuff.

Yeti is one of the CT logs (and Yeti2022 the 2022 shard of it, containing certs that expire in 2022). CT logs are independent of each other, there is not really a "global log", although the monitoring/search sites aggregate the data from all logs. Each certificate is added to multiple logs, so the loss of one doesn't cause a problem for the certs in it. (Maybe it's also possible to still trust Yeti2022 for the parts of the log that are well-formed, which would decrease the number of impacted certs even more, not familiar enough with the implementations for that)
Post reply on HN