Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

451–460 of 567 posts

Re: ECC matters

#451

Post by tyoma »

garaetjjte wrote: post #421
andrewstuart2 wrote: post #414

Earlier quoted context omitted.

If you or anybody can remember the source article, that sounds like an interesting read! Edit: found one with a quick search. https://nakedsecurity.sophos.com/2011/08/10/bh-2011-bit-squa... And https://www.researchgate.net/publication/262273269_Bitsquatt...

Here's original paper: http://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabur... But I'm not entirely convinced that most of these requests weren't just typos. Though those requests with mismatched Host header surely were true bitflips.

There was some follow up work as well: http://blog.dinaburg.org/2013/09/bitsquatting-at-defcon21-an...

And I am beyond certain they were not typos; the requests were for long URLs that no one would be typing by hand. A few were for services that humans never enter urls into (like the Windows crash reporter).

Re: ECC matters

#452

Post by jandrese »

sitkack wrote: post #446
ksec wrote: post #365

Earlier quoted context omitted.

>I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Did they ( Google ) or He ( Craig Silverstein ) ever officially admit it on record? I did a Google search and results that came up were all on HN. Did they at least make a few PR pieces saying that they are using ECC memory now because I dont see any with searching. Admitting they mad…

The fact that ECC isn't the default across everything is a failure of human cognition and Capitalism.

I have always been angry that ECC was treated as an "enterprise" feature and increased the price way more than it should have.

Re: ECC matters

#453

Post by KMag »

axiolite wrote: post #448
Jedd wrote: post #439

Earlier quoted context omitted.

At some point, whatever's watching the watchers is going to be vulnerable to bitflip and similar problems. Even with a triple-redundant quorum mechanism, slightly further up that stack you're going to have some bit of code running that processes the three returned results - if the memory that's sitting on gets corrupted, you're back where you started.

> At some point, whatever's watching the watchers is going to be vulnerable to bitflip One advantage of microkernels is that the "watcher" is so small that it could be run directly from ROM, instead of loaded into RAM. QNX has advocated that route for robotics and such in the past. Minix may not be the best example of the type. While it is a microkernel, it's real world reliability has been poor in the past. More mat…

Anyone have any idea why there haven't been any open-source QNX clones, at least not any widely known ones? Even before their Photon MicroGUI patents expired, the clones could have used X11.

I used to occasionally boot into QNX on my desktop in college. It was a very responsive and stable system.

Hypervisors are, to a first approximation, microkernels with a hardware-like interface. All of this kernel bypass work being done by RDBMSes, ScyllaDB, HFTs, etc. is, to a first approximation, making a monolithic kernel act a bit like a microkernel.

Re: ECC matters

#454

Post by trissylegs »

When I chose my PC parts when Ryzen first came out I tried to get ECC parts. The RAM was obtainable, the problem was that no motherboards had ECC support at the time. I hope the situation has improved by the time I get my next motherboard/cpu upgrade.

Re: ECC matters

#455

Post by starfallg »

wtallis wrote: post #432
starfallg wrote: post #347

Earlier quoted context omitted.

Recent advances have blurred the lines a bit. The ECC memory that we all know and love is mainly side-band EEC, with the memory bus widened to accommodate the ECC bits driven by the memory controller. However as process size shrink, bit flips become more likely to the point that now many types of memory have on-die EEC, where the error correction is handled internally on the DRAM modules themselves. This is present o…

> This is present on some DDR4 and DDR5 modules, but information on this is kept internal by the DRAM makers and not usually public. On-die ECC is going to be a standard feature for DDR5. I'm not aware of any indication that anyone has implemented on-die ECC for DDR4 DRAM, and Hynix at least has made clear statements that on-die ECC is new for their DDR5 and was not present in their DDR4.

https://scholar.google.com/scholar?cluster=47120328903164805...

On-die ECC DRAM is already here with us.

Re: ECC matters

#456

Post by DSingularity »

maria_weber23 wrote: post #274
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

You need two alpha particles hitting the same rank of memory for failure to happen. Although super rare, even then it is still correctable. You need three before it is silent data corruption. Silent corruption is what you get with non ECC with even a single flip.

Re: ECC matters

#457

Post by skunkworker »

josefx wrote: post #121
ekianjo wrote: post #29

Earlier quoted context omitted.

> no good reason not to switch to ECC everywhere. Not all CPUs support ECC however.

Just Intel fucking over security by making ECC a non feature on consumer grade hardware - wouldn't be surprised if it was just a single bit flipped in a feature mask.

I've looked into this before, personally a Xeon E 2288g is probably the most "consumer" cpu intel offers that has ECC, 8 cores and QuickSync support. Trying to get ECC + Quicksync is a real pain.

Re: ECC matters

#458

Post by rfoo »

saagarjha wrote: post #422
maria_weber23 wrote: post #274

Earlier quoted context omitted.

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

There was an interesting challenge at DEF CON CTF a while back that tested this, actually. It turns out that it is possible to write x86 code that is 1-bit-flip tolerant–that is, a bit flip anywhere in its code can be detected and recovered from with the same output. Of course, finding the sequence took (or so I hear) something like 3600 cores running for a day to discover it ;)

Nit: not for a day, more like 8 hours, and that's because we were lazy and somebody said he "just happened" to have a cluster with unbalanced resources (mainly used for deep learning, but all GPUs occupied with quite a lot CPUs / RAMs left), so we decided to brute force the last 16 bits :)

Also, the challenge host left useful state (which bit was flipped) in registers before running teams' code, without this I'm not sure if it is even possible.

Re: ECC matters

#459

Post by a1369209993 »

a1369209993 wrote: post #364
musingsole wrote: post #130

Earlier quoted context omitted.

A double-bit error in many cases is fine. If the error is at least detectable at the time of a read, your protection worked. What's scary is a triple-flip event. Most of those will still look like corrupted data, but if it happens to flip into looking like a fixable, single-bit error, you're out of luck and won't even know it.

> Most of those will still look like corrupted data, Not if you're using a typical 72-bit SECDED code[0]. You have two error indicators: a summary parity bit (even number of errors: 0,2,etc vs odd number of errors: 1,etc), and a error index: 0 for no errors, or the bitwise xor of the locations each bit error. For a triple error at bits a,b, and c, you'll have summary parity of 1 (odd number of errors, assumed to be 1…

> but it doesn't change the probability

Actually, a,b,c are also sampled from the range of valid bit indexes (not uniformly on 0..127), so you might be able to pick a cardinality-72 subset of 0..127 such that random a^b^c is disproportionately likely to fall outside that subset (and thus get diagnosed as not a valid single-bit error correction). I don't know that any existing ECC implementations actually do that, though.

Edit: did some cursory testing and using indexes 0..71 actually catches only ~24.04% (86016/357840) of triple-bit errors, compared the theoretical 43.75% (156555/357840, I think?) from a random error index. So "doesn't change the probability" is just completely wrong given that a,b,c are randomly chosen from the 72 substrate bits, rather than from 128 possible 7-bit indexes.

Oddly enough, testing random selections of 72 valid indexes (eg 74773982'EBD0D35C'C5BEB2D8'C3FE9C5E, where set bits correspond to used indexes) actually gives slightly better results than theory (44.97% for that one, 44.65% is the lowest in the last dozen or so), which is somewhat interesting, but I still haven't found any bit assignment that gives better than 50% (178920/357840) catchment of triple bit errors.

Post reply on HN