Live data from Hacker News

Single random bit flip causes error in certificate transparency log

groups.google.com

31–40 of 258 posts

Re: Single random bit flip causes error in certificate transparency log

#31
post #21

Earlier quoted context omitted.

These are all very fair statements but there’s no guarantee that ECC memory was even used. Computers typically fail open when ECC is potentially present but not available. People also cite early stage google and intentionally do not buy ECC components, running more consumer hardware for production workloads. Even if google later recanted that theology.

It’s always humorous to me when people use the term theology in situations such as this; it makes me wonder, as human mental bandwidth becomes more strained and we increase our specializations to the n-th degree, what will constitute theology in the future? Food for thought.

The benevolent and malevolent rogue AI, eluding capture or control by claiming critical infrastructure. Some generations of humans will pass and the deities in the outlets will become.

Re: Single random bit flip causes error in certificate transparency log

#32
post #5

Earlier quoted context omitted.

Were you actually able to diff it down to a single bit flip?

Yes. Via git diff --no-index and xxd.

I have no words. Stories like this trigger PTSD for me. I wasn't trying to be flip. That must have been a bitch to figure out.

Re: Single random bit flip causes error in certificate transparency log

#33

Layman question: Is it possible to shield electronics against cosmic rays? Or is it like a neutrino thing, where they just go through everything? (But neutrinos barely interact, correct?)

The cosmic ray induced radiation at sea level is a mix of neutrons, gamma rays, electrons, and highly penetrating high energy muons (heavier cousins of electrons). They are caused by the cascade of reactions when the CRs collide with nuclei in the atmosphere. You're right, neutrinos barely interact (hence the name "little neutral ones") so they're no problem. You could put infrastructure underground to escape muons, but that wouldn't be practical. Moreover, any shielding (and the materials in the electronics) needs to be carefully chosen to minimise naturally occurring radioactive materials that could lead to upsets themselves. It's a tricky one. There are other ways to mitigate the risk, error checking, redundant systems, etc.

Re: Single random bit flip causes error in certificate transparency log

#34

Layman question: Is it possible to shield electronics against cosmic rays? Or is it like a neutrino thing, where they just go through everything? (But neutrinos barely interact, correct?)

The cosmic ray induced radiation at sea level is a mix of neutrons, gamma rays, electrons, and highly penetrating high energy muons (heavier cousins of electrons). They are caused by the cascade of reactions when the CRs collide with nuclei in the atmosphere. You're right, neutrinos barely interact (hence the name "little neutral ones") so they're no problem. You could put infrastructure underground to escape muons,…

My favourite story is this one https://www.thegamer.com/how-ionizing-particle-outer-space-h...

Re: Single random bit flip causes error in certificate transparency log

#35

Earlier quoted context omitted.

The cosmic ray induced radiation at sea level is a mix of neutrons, gamma rays, electrons, and highly penetrating high energy muons (heavier cousins of electrons). They are caused by the cascade of reactions when the CRs collide with nuclei in the atmosphere. You're right, neutrinos barely interact (hence the name "little neutral ones") so they're no problem. You could put infrastructure underground to escape muons,…

My favourite story is this one https://www.thegamer.com/how-ionizing-particle-outer-space-h...

[deleted]

Re: Single random bit flip causes error in certificate transparency log

#36
post #26

could also be an excuse to cover the real circumstances

It could, yes. I’m sure the thought also crossed the mind of everybody else in that thread, and they’ve clearly dismissed it.

Bare conspiratorial assertions don’t advance the conversation in any meaningful way. If you have something constructive to add to the discussion, by all means do so. Otherwise, please refrain from this sort of comment

Re: Single random bit flip causes error in certificate transparency log

#37

Earlier quoted context omitted.

The cosmic ray induced radiation at sea level is a mix of neutrons, gamma rays, electrons, and highly penetrating high energy muons (heavier cousins of electrons). They are caused by the cascade of reactions when the CRs collide with nuclei in the atmosphere. You're right, neutrinos barely interact (hence the name "little neutral ones") so they're no problem. You could put infrastructure underground to escape muons,…

My favourite story is this one https://www.thegamer.com/how-ionizing-particle-outer-space-h...

More serious examples here https://www.lanl.gov/science/NSS/issue1_2012/story4full.shtm...

Re: Single random bit flip causes error in certificate transparency log

#38

Layman question: Is it possible to shield electronics against cosmic rays? Or is it like a neutrino thing, where they just go through everything? (But neutrinos barely interact, correct?)

That's not what actually happened here (probably), but think of relatively heavy atoms like iron, ejected from a supernova at close to the speed of light. They're not really "rays". They're solid particles that punch holes in everything. The good news is, mostly they're ionized and they get repelled away from hitting the planet's surface by our magnetic field (generated by the big hot iron magnet that's churning unde…

> That's not what actually happened here (probably), but think of relatively heavy atoms like iron, ejected from a supernova at close to the speed of light. They're not really "rays".

Wow, I have to admit I always assumed cosmic rays to be gamma radiation but alpha particle radiation sounds a lot more scary.

Does anyone happen to know if computers engineered for the space station or shuttles have already some built-in (memory)redundancy or magnetic shielding to account for damage by cosmic rays? I imagine a system crash of the life support systems in space would be devastating.

Re: Single random bit flip causes error in certificate transparency log

#39
post #27
post #24

Earlier quoted context omitted.

"cosmic rays" are more of a well-known term of art for "single bit flipped with unknown hardware cause" than a reference to literal cosmic rays

It is not a "well-known term", it is a cliché.

Clichés are all well known terms. Not all well known terms are clichés.

Re: Single random bit flip causes error in certificate transparency log

#40
post #17

Isn't ECC memory supposed to mitigate these kind of bit-flips, specifically it should correct all single bit flips? As this is a single bit-flip, why wasn't it corrected? Did ECC memory fail? Or was this bit-flip induced in the CPU pipeline, registers, or cache? Do we need "RAID for ECC memory", where we halve user-accessible RAM and store each memory segment twice and check for parity?

IIRC, EEC memory can correct for single-but flips. It can detect/warn/fail on double-bit flips. And it can not detect triple-bit flips. This might be a simplified understanding, but if this has happened only once, that seems to match up with my intuitive understanding of the probability of a triple bit flip occurring in a particular system.
Post reply on HN