Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

151–160 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#151

Do nuclear missiles have this anywhere? bool launch = false; if(launch) ...

This is why any system that can cause physical harm should have hardware interlocks. A computer error can be bypassed with a physical switch.

For something like nuclear missile launch control you have redundant systems and hardware interlocks that require human intervention to launch.

Re: Single random bit flip causes error in certificate transparency log

#152
post #140
post #68

Earlier quoted context omitted.

Public CAs are not that type of people; I would be disapointed if that were not running two seperate systems checking each other for consistancy; having top of the range ECC running well inside its specification must be table stakes.

> Public CAs are not that type of people I think you hold public CAs to a higher standard than many hold themselves to. There are hundreds of CAs and many (if not most) are shockingly awful. Which is why we have had a huge push back against the PKI cartels.

Not hundreds. There are currently 52 root CA operators trusted by Mozilla (and thus Firefox, but also most Linux systems and lots of other stuff) a few more are trusted only by Microsoft or Apple, but not hundreds.

But also, in this context we aren't talking about the CAs anyway, but the Log operators, and so for them reliability is about staying qualified, as otherwise their service is pointless. There are far fewer of those, about half-a-dozen total. Cloudflare, Google, Digicert, Sectigo, ISRG (Let's Encrypt), and Trust Asia.

[Edited, I counted a column header, 53 rows minus 1 header = 52]

Re: Single random bit flip causes error in certificate transparency log

#154
post #123
post #110

Earlier quoted context omitted.

> The push for crypto without ecc ram is a nonstop horror show That's a bit hyperbolic. First, ECC doesn't protect the full data chain, you can have a bitflip in a hardware flip flop (or latch open a gate that drains a line, etc...) before the value reaches the memory. Logic is known to glitch too. Second: ECC is mostly designed to protect long term storage in DRAM. Recognize that a cert like this is a very short-ter…

If a system is critical it should run on multiple machines in multiple locations and "sync with checks" kinda like the oh so hated and totally useless blockchains. Then if such a bit-flip would occur it would never occur on all machines at the same time in the same data. And on top of that you could easy make the system fix itself if something like that happens (simply assume the majority of nodes didn't have the bit…

I haven't thought about whether an actual blockchain is really the best solution, but the redundancy argument is legitimate. We've been doing it for decades in other systems where an unnoticed bit flip results in complete mission failure, such as an Apollo mission crash.

I'm not really sure what Yeti 2022 is exactly, so take this with heaps of salt, but it seems like this is a "mission failure" event -- it can no longer continue, except as read only. Crypto systems, even more than physical systems like rockets, suffer from such failures after just "one false step". Is the cost of this complete failure so low that it doesn't merit ECC? Extremely doubtful. Is it so low that it doesn't merit redundancy? More open for debate, but plausibly not.

I know rockets experience more cosmic rays and their failure can result in loss of life and (less importantly) losing a lot more money, and everything is a tradeoff -- so I'm not saying the case for redundancy is water tight. But it's legitimate to point out there is an inherent and it seems under-acknowledged fragility in non-redundant crypto systems.

Re: Single random bit flip causes error in certificate transparency log

#155
post #122

Earlier quoted context omitted.

I've always wondered about that. Seems to me that if you can shield one side from the sun's heat and expose the other side to the cold of space, an MRI magnet's superconductors should be quite happy with the temperature. A big ol' magnet would be a pain to charge up once, and then provide long-term shielding.

MRIs are big electromagnets and they use big power and produce big heat. In space you have no convection so you must radiate away all of your heat which is challenging. Maybe you can make something better with superconductors that doesn't use much power, but I don't think it exists yet.

Superconductors famously don’t produce heat. Putting a superconducting magnet in shade (though that includes from the Earth not just from Sol) will keep it cool once it gets cool.

This is what the James Webb is going to be doing, though for non-superconducting reasons.

Re: Single random bit flip causes error in certificate transparency log

#156

Are “cosmic rays” actually the only or primary way bits get flipped? Or is it just a stand-in for “all the ways non-ECC RAM can spontaneously have an erroneous bit or two”?

Faulty RAM can produce errors and it's hard to catch those, even memtest might not detect it. I'm not sure if non-faulty non-ECC RAM can spontaneously have errors, but you can't be sure, cosmic rays are real, unless you've put your PC into a thick lead case, LoL.

In my experience with ECC RAM on couple thousand servers over a few years, we had a couple machines throw one ECC correctable error and never have an issue again.

It seemed more common for a system to move from no errors to a consistent rate; some moved to one error per day, some 10/day, some to thousands per second which kills performance because of machine check exception handling.

The one off errors could be cosmic rays or faulty ram or voltage droop or who knows what, the repeatable errors are probably faulty ram, replacing the ram resolved the problem.

Re: Single random bit flip causes error in certificate transparency log

#157
post #67

Cosmic-ray bit flipping is real and it has real security concerns. This also makes Intel's efforts at market segmentation by not having ECC support in any consumer CPUs [1] even more unforgivable and dangerous. Example: bitsquatting on domains [2]. [1]: https://arstechnica.com/gadgets/2021/01/linus-torvalds-blame... [2]: https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...

AMD is also doing market segmentation on their APU series of Ryzen. PRO vs. non PRO.

It should also be mentioned that Ryzen is a consumer CPU and you're stuck (mostly) with consumer motherboards, none of which tell you the level of ECC support they provide. Some motherboards do nothing with ECC! Yes, they "work" with it. But that means nothing. Motherboards need to say they correct single bit errors and detect double bit errors.[1] None of the Ryzen motherboards say this. Not a single one that I could find.

Maybe Asrock Rack, but that's a workstation/server motherboard. Which is also going for $400-600. You think that $50 Gigabyte motherboard is doing the right thing regarding ECC? That's a ton of faith right there.

Consumer Ryzen CPUs may support ECC, but that's meaningless without motherboards testing it and documenting their support of it. So no, Ryzen really does not support ECC if you ask me.

[1] https://cr.yp.to/hardware/ecc.html

Re: Single random bit flip causes error in certificate transparency log

#158
post #124
post #110

Earlier quoted context omitted.

> The push for crypto without ecc ram is a nonstop horror show That's a bit hyperbolic. First, ECC doesn't protect the full data chain, you can have a bitflip in a hardware flip flop (or latch open a gate that drains a line, etc...) before the value reaches the memory. Logic is known to glitch too. Second: ECC is mostly designed to protect long term storage in DRAM. Recognize that a cert like this is a very short-ter…

This is just learned helplessness because Intel were stingy as shit for over a decade and wanted to segregate their product lines. Error correction is literally prevalent in every single part of every PHY layer in a modern stack, it is an absolute must, and the lack of error correction in RAM is, without question, a ridiculous gap that should have never been allowed in the first place in any modern machine, especiall…

> By definition, if a failure occurs, it's because it passed multiple safeguards that were already in place.

Having worked on a good bunch of critical system, there aren't multiple safeguards in most hardware.

E.g. a multiplication error in a core will not be detected by an external device. Or a bit flip when reading cache, or from a storage device.

Very often the only real safeguard is to do the whole computation twice on two different hosts. I would rather have many low-reliability hosts and do computation twice than few high-reliability and very expensive host.

Unfortunately the software side is really lagging behind when it comes to reproducible computing. Reproducible builds are a good step in that direction and it took many decades to get there.

Re: Single random bit flip causes error in certificate transparency log

#159
post #67

Cosmic-ray bit flipping is real and it has real security concerns. This also makes Intel's efforts at market segmentation by not having ECC support in any consumer CPUs [1] even more unforgivable and dangerous. Example: bitsquatting on domains [2]. [1]: https://arstechnica.com/gadgets/2021/01/linus-torvalds-blame... [2]: https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa...

AMD is also doing market segmentation on their APU series of Ryzen. PRO vs. non PRO. It should also be mentioned that Ryzen is a consumer CPU and you're stuck (mostly) with consumer motherboards, none of which tell you the level of ECC support they provide. Some motherboards do nothing with ECC! Yes, they "work" with it. But that means nothing. Motherboards need to say they correct single bit errors and detect double…

DDR5 has a modicum of ECC so things might slowly improve. Maybe DDR6 will be full ECC and we will no longer have this market segmentation in the 2030s. Wow, that’s a long time though.

PS: why didn’t apple do the right thing with the M1? My guess is the availability of the memory which again points to changing it at the memory spec level.

Re: Single random bit flip causes error in certificate transparency log

#160
post #112

Earlier quoted context omitted.

Checksumming filesystems and file transfer protocols cover many cases. SCP, rsync, btrfs, and zfs all fix this problem. As for guaranteeing the computed data is correct: I know space systems often have two redundant computers that calculate everything and compare results. It's crazy expensive and power demanding, but it all but solves the problem.

If they have two computers and results differ, how do they decide which one is correct ? ;-)

Usually they have an odd number. The Space Shuttle had five, of which the fifth was running completely different software. In case of a 2/2 split the crew could shut down a pair (in case the failure was clear) or switch to the backup computer.
Post reply on HN