Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

Post reply on HN
Print view View original

71–80 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#71

Post by MayeulC »

joshspankit wrote: post #59
noduerme wrote: post #11

Earlier quoted context omitted.

That's not what actually happened here (probably), but think of relatively heavy atoms like iron, ejected from a supernova at close to the speed of light. They're not really "rays". They're solid particles that punch holes in everything. The good news is, mostly they're ionized and they get repelled away from hitting the planet's surface by our magnetic field (generated by the big hot iron magnet that's churning unde…

I had thought that Starlink would become extremely compelling when the servers were in orbit as well, but maybe that’s naive. Cubesats with massive arrays of active storage might be far too difficult (aka costly) to protect properly.

One of the issues with putting servers in orbit is cooling; you can't just use fans in space. On the other hand, real estate is pretty cheap. Servicing is hard, though, and micrometeorites are another risk. Plus launch costs being high, and radiation an issue, I don't see it happening any time soon outside of very specific areas.

Re: Single random bit flip causes error in certificate transparency log

#72

Post by [deleted] »

myself248 wrote: post #56
ben_w wrote: post #47

Earlier quoted context omitted.

Radiation hardening is basically everything you can manage with the weight limit: https://en.wikipedia.org/wiki/Radiation_hardening IIRC nobody is currently using magnetic fields for shielding, I don’t know if that’s due to insufficient effectiveness, power consumption, or unwanted interactions e.g. with Earth’s magnetosphere.

I've always wondered about that. Seems to me that if you can shield one side from the sun's heat and expose the other side to the cold of space, an MRI magnet's superconductors should be quite happy with the temperature. A big ol' magnet would be a pain to charge up once, and then provide long-term shielding.

[deleted]

Re: Single random bit flip causes error in certificate transparency log

#73

Post by detaro »

oconnor663 wrote: post #70
tsegratis wrote: post #45

Do nuclear missiles have this anywhere? bool launch = false; if(launch) ...

I don't know about nuclear missiles, but on the Space Shuttle I think they had four duplicate flight computers, and the outputs of all of them would be compared to look for errors. (They also had a fifth computer running entirely different software, as a failover option.)

The Space Shuttle also had a HP-41C calculator with special software to help with manual flying in case of a general computer failure: https://airandspace.si.edu/collection-objects/calculator-han...

I wonder how viable that was for different stages in the flight.

Re: Single random bit flip causes error in certificate transparency log

#74

Post by MayeulC »

Leparamour wrote: post #41
Kimitri wrote: post #14

Earlier quoted context omitted.

It may be a regional thing but I have never heard ”bit rot” refer to legacy code. In the retro computing circles bit rot refers to hardware defects (usually floppies or other storage media) caused by cosmic rays or other environmental hazards.

I have to agree with Kimitri here. This is the only context in which I have ever encountered the term 'bit rot'.

I agree this is the primary context, but I've seen unmaintained (or very old) software being reffered to as "bit rotting" by extension. As in, forward compatibility might break due to obsolete dependencies, etc.

Re: Single random bit flip causes error in certificate transparency log

#75

Post by egberts1 »

Bit flip is the bane of satellite communication, especially if you tried to use FTP over it.

Also, for critical eon-duration record keeping, run-length encoding or variable record size are harder to maintain and recover from large file on ROM-type storages than fixed-length record or text-type (i.e. ASCII log, or JSON) … against multiple cosmic-type alterations.

Sure that you could do max. Shannon approach for multi-bit ECC, but text-based will be recovered a lot quicker (given then-unknown formatting).

Re: Single random bit flip causes error in certificate transparency log

#77

Post by fennecfoxen »

joshspankit wrote: post #59
noduerme wrote: post #11

Earlier quoted context omitted.

That's not what actually happened here (probably), but think of relatively heavy atoms like iron, ejected from a supernova at close to the speed of light. They're not really "rays". They're solid particles that punch holes in everything. The good news is, mostly they're ionized and they get repelled away from hitting the planet's surface by our magnetic field (generated by the big hot iron magnet that's churning unde…

I had thought that Starlink would become extremely compelling when the servers were in orbit as well, but maybe that’s naive. Cubesats with massive arrays of active storage might be far too difficult (aka costly) to protect properly.

You can’t just put an SSD or whatever in orbit and expect reasonable read latencies at all times. It’s in orbit. It moves. Half the time it’s on the wrong side of the planet.

Re: Single random bit flip causes error in certificate transparency log

#78

Post by xoa »

joshspankit wrote: post #62
layoutIfNeeded wrote: post #19

Earlier quoted context omitted.

It was a MacStadium ( https://www.macstadium.com ) build server, so most likely non-ECC.

Strange that build services are not just going full ECC, especially with cheaper hardware now supporting it.

Says it's a Mac/iOS build platform. Since it's a commercial service they're probably complying with the license and thus using actual Mac hardware, and in turn the only ECC option is the really awful value, outdated Mac Pro. Seems more likely they're using Minis instead, or at least mostly Minis. An unfortunate thing about Apple hardware (says someone still nursing along a final 5,1 Mac Pro for a last few weeks).

Re: Single random bit flip causes error in certificate transparency log

#79

Post by noduerme »

layoutIfNeeded wrote: post #64
noduerme wrote: post #32

Earlier quoted context omitted.

I have no words. Stories like this trigger PTSD for me. I wasn't trying to be flip. That must have been a bitch to figure out.

Well, it wasn't that hard to uncover actually. We knew that the same build succeeds on our machines. So we only had to find what the difference was between the two :) As Arthur Conan Doyle put it: "Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth." ¯\_(ツ)_/¯

Well done, Watson. This calls for a bit of snuff. Seriously, this is the kind of thing that keeps me up at night, and it's nice to hear a happy ending =D

Re: Single random bit flip causes error in certificate transparency log

#80

Post by namibj »

formerly_proven wrote: post #43
fulafel wrote: post #30

Earlier quoted context omitted.

These things exist, using trade names like chipkill or lockstep memory. Though they don't need to sacrifice half of the memory chips to get good error recovery properties. Note that this is still not end-to-end protection of data integrity. Bit flips happen in networking, storage, buses between everything, caches, CPUs, etc. See eg [1] [1] https://arxiv.org/abs/2102.11245 Silent Data Corruptions at Scale (based on em…

According to the Intel developer's manual L1 has parity and all caches up from that have ECC. This would seem to imply that the ring / mesh also has at least parity (to retry on error). Parity instead of ECC on L1(D) makes sense since the L1(D) has to handle small writes well, while the other caches deal in lines.

Zen3 has parity on L1I and ECC on L1D and beyond. Even ECC on consumer hardware (it's just a few extra traces on the motherboard).
Post reply on HN