Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

481–490 of 567 posts

Re: ECC matters

#481

Post by fragmede »

finiteloop wrote: post #477
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

One of the best quotes in the Google quotes file an early Googler maintained (I am sure I am screwing it up): “I’ve heard of defensive programming, but never adversarial memory.” — Ben Gomes

Close!

> I've never thought of defensive programming in terms of adversarial memory.

Re: ECC matters

#482

Post by shrewduser »

sitkack wrote: post #446
ksec wrote: post #365

Earlier quoted context omitted.

>I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Did they ( Google ) or He ( Craig Silverstein ) ever officially admit it on record? I did a Google search and results that came up were all on HN. Did they at least make a few PR pieces saying that they are using ECC memory now because I dont see any with searching. Admitting they mad…

The fact that ECC isn't the default across everything is a failure of human cognition and Capitalism.

[flagged]

Re: ECC matters

#483

Post by dannyw »

defanor wrote: post #66
loeg wrote: post #33

Earlier quoted context omitted.

Yeah, it's real obnoxious of Intel to silo ECC support off into the Xeon line, isn't it? I switched to ECC memory in 2013 or 2014 with a Xeon E3 (fundamentally a Core i7 without the ECC support fused off) and of course a Xeon-supporting motherboard (with weird "server board" quirks: e.g., no on-board sound device). I love that AMD doesn't intentionally break ECC on its consumer desktop platforms and upgraded to the T…

I've considered using an AMD CPU instead of Intel's Xeon on the primary desktop computer, but even low-end Ryzen Threadripper CPUs have TDP of 180W, which is a bit higher than I'd like. And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those, so one won't be able to return/replace a CPU if it doesn't work with ECC memory, AIUI, making it risky. Though I don't know how common i…

You can use BIOS settings to change the TDP to whatever you like, with substantially higher efficiency if you are under-powering and substantially lower efficiency if you are over-powering.

Re: ECC matters

#484

Post by dannyw »

DCKing wrote: post #321
CalChris wrote: post #262

Earlier quoted context omitted.

Yeah, the iMac Pro has the Xeon W and ECC. T'would be nice if the Apple Silicon MacBook Pro had it. There's not much of a reason to pay for the Pro over the Air. But like Linus, I'm going to blame Intel for this situation in the market. Maybe Apple will strike out on its own with Apple Silicon but since their dominant use case is phones, I'll not hold my breath.

Unless something weird happens, the next generation of the Apple M-line will use LPDDR5 memory instead of the LPDDR4X used in the Apple M1. While it probably won't support error correction monitoring , LPDDR5 has built in error correction that silently corrects single bit flips. That alone should be a huge reliability improvement. LPDDR5 will enable some much needed level of error correction in a metric ton of other…

Will this also exist for consumer DDR5?

Re: ECC matters

#485

Post by klodolph »

shrewduser wrote: post #482
sitkack wrote: post #446

Earlier quoted context omitted.

The fact that ECC isn't the default across everything is a failure of human cognition and Capitalism.

[flagged]

Just because capitalism fails to provide something does not mean that communism would solve the problem, or that criticism of capitalism as a system is uncalled for. This is not some kind of binary capitalism/communism world we live in.

Re: ECC matters

#486

Post by dannyw »

adrian_b wrote: post #331
incrudible wrote: post #149

Earlier quoted context omitted.

When you say bitflips were "common" on thousands of physical machines, does that mean you observed thousands of bitflips? Otherwise, I would think that an unlikely event becoming 1000x more likely by sheer numbers would have warped your perception. I believe that hardware reliability is mostly irrelevant, because software reliability is already far worse. It doesn't matter whether a bitflip (unlikely) or some bug (li…

On a single computer with a large memory, e.g. 32 GB or more, the time between errors can be of a few months, if you are lucky to have good modules. Moreover, some of the errors will have no effect, if they happened to affect free memory. Nevertheless, anyone who uses the computer for anything else besides games or movie watching, will greatly benefit from having ECC memory, because that is the only way to learn when…

Plus, ECC RAM is so accessible these days thanks to Ryzen. All the asrock mobos, from the lowest end to the highest end, advertise official support for it.

Re: ECC matters

#487

Post by dannyw »

Springcleaning wrote: post #323
cbanek wrote: post #77

As someone who has had to read thousands of random game crash reports from all over the interwebs (you know when Windows says you might want to send that crash log? like that), I totally agree. Of all the things to be worried about, like OS bugs, bad hardware configuration, etc. bad memory is one of those really troubling things. You look at the code and say "it's can't make it here, because this was set" but when yo…

Worse than a game crash is your data. It is incomprehensible that there are still NAS devices being sold without ECC support. Synology took a step in the right direction to offer prosumer devices with ECC but it is not really advertised as such. It is actually difficult to find which do have ECC and which ones don't.

Blame Intel for that. They dropped support for ECC RAM on the new Celerons and i3s.

Re: ECC matters

#488

Post by dannyw »

tokamak-teapot wrote: post #52
simias wrote: post #9

I used to be pretty skeptical of ECC for consumer-grade hardware, mainly because I felt that I'd always prefer cheaper/more RAM over ECC RAM even if it meant that I'd get a couple of crash every year due to rogue bitflips. For servers it's a different story, but for a desktop I'm fine dealing with some instability for better performance. But these days with the RAM density being so high and bitflipping attacks being…

Are there any Ryzen boards that support ECC and actually correct errors ?

Yes, almost all of them correct single-bit flips and detect but do not correct multiple hit flips.

Re: ECC matters

#489

Post by klodolph »

DSingularity wrote: post #456
maria_weber23 wrote: post #274

Earlier quoted context omitted.

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

You need two alpha particles hitting the same rank of memory for failure to happen. Although super rare, even then it is still correctable. You need three before it is silent data corruption. Silent corruption is what you get with non ECC with even a single flip.

Where are you getting this from? My understanding is that these errors are predominantly caused by secondary particles from cosmic rays hitting individual memory cells, and I've never heard something so precise as "you need two alpha particles". Aren't the capacitances in modern DRAM chips extremely small?

Re: ECC matters

#490

Post by saagarjha »

rfoo wrote: post #458
saagarjha wrote: post #422

Earlier quoted context omitted.

There was an interesting challenge at DEF CON CTF a while back that tested this, actually. It turns out that it is possible to write x86 code that is 1-bit-flip tolerant–that is, a bit flip anywhere in its code can be detected and recovered from with the same output. Of course, finding the sequence took (or so I hear) something like 3600 cores running for a day to discover it ;)

Nit: not for a day, more like 8 hours, and that's because we were lazy and somebody said he "just happened" to have a cluster with unbalanced resources (mainly used for deep learning, but all GPUs occupied with quite a lot CPUs / RAMs left), so we decided to brute force the last 16 bits :) Also, the challenge host left useful state (which bit was flipped) in registers before running teams' code, without this I'm not…

Sure, all's fair in a CTF. That story came to me through the mouths of at least a handful of people, who might have a bit of an incentive to exaggerate given that they hadn't quite been able to get to zero and might be a just a little sour :P

The state was quite helpful, yes–for x86 it seems like a "clean slate" shellcode would be quite difficult, if impossible, to achieve as we saw. However, I am left wondering how other ISAs would fare…perhaps worse, since x86 is notoriously dense. But maybe not? The fixed-width ones would probably be easy to try out, at least.

Post reply on HN