Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

311–320 of 567 posts

Re: ECC matters

#311

Post by colejohnson66 »

maria_weber23 wrote: post #274
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

> You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum.

The Apollo missions (or was it the Space Shuttle?) did this. They had redundant computers that would work with each other to determine the “true” answer.

Re: ECC matters

#312

Post by colejohnson66 »

gigatexal wrote: post #290
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

I mean early on sure at a startup where you’re not printing money I can see how saving on hardware makes sense. But surely you don’t need an MBA to know that hardware will continue to get cheaper whereas developers and their time will only get more expensive: better to let the hardware deal with it than to burden developers with it ... I’d have made the case for ECC but hindsight being what it is ...

But if you can save $1M+ now, then throw the cost of fixing it onto the person who replaces you, why do you care? You already got your bonus and jumped ship.

Re: ECC matters

#313

Post by colejohnson66 »

theevilsharpie wrote: post #186
defanor wrote: post #66

Earlier quoted context omitted.

I've considered using an AMD CPU instead of Intel's Xeon on the primary desktop computer, but even low-end Ryzen Threadripper CPUs have TDP of 180W, which is a bit higher than I'd like. And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those, so one won't be able to return/replace a CPU if it doesn't work with ECC memory, AIUI, making it risky. Though I don't know how common i…

> And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those ECC isn't validated by AMD for AM4 Ryzen models, but it's present and supported if the motherboard also supports it. Many motherboards have ECC support (the manual will say for sure), and a handful of models even explicitly advertise it as a feature. I have a Ryzen 9 3900X on an ASRock B450M Pro4 and 64 GB of ECC DRAM,…

What do you mean by “validated”? There’s the silicon, but they don’t test it?

Re: ECC matters

#314

Post by rajesh-s »

GregarianChild wrote: post #304
rajesh-s wrote: post #301

Earlier quoted context omitted.

Right! Section 1.3 of this publication discusses possible mitigations for the row hammer problem and where ECC fits in https://users.ece.cmu.edu/~omutlu/pub/rowhammer-summary.pdf

The paper you cite is from 2014 and the mitigations discussed there have all been circumvented. [1] is from 2020 and a better read for Rowhammer mitigation. [1] J. S. Kim et al, Revisiting RowHammer: An Experimental Analysis of Modern DRAM Devices and Mitigation Techniques https://arxiv.org/abs/2005.13121

Thanks for pointing that out!

Re: ECC matters

#315

Post by haolez »

colejohnson66 wrote: post #311
maria_weber23 wrote: post #274

Earlier quoted context omitted.

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

> You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. The Apollo missions (or was it the Space Shuttle?) did this. They had redundant computers that would work with each other to determine the “true” answer.

Sounds similar to smart contracts running on a blockchain :)

Re: ECC matters

#316

Post by gsvelto »

jeffbee wrote: post #204
core-questions wrote: post #164

Earlier quoted context omitted.

https://github.com/netdata/netdata/issues/1508 Looks like `mcelog --client` might be a starting place? Feed that into your metrics pipeline and alert on it like anything else...

Newer linux have replaced mcelog with edac-util. I think most shops operating systems at that scale are getting their ECC errors out of band with IPMI SEL, though.

It's rasdaemon these days: https://www.setphaserstostun.org/posts/monitoring-ecc-memory...

Re: ECC matters

#317

Post by colejohnson66 »

marcosdumay wrote: post #72
defanor wrote: post #66

Earlier quoted context omitted.

I've considered using an AMD CPU instead of Intel's Xeon on the primary desktop computer, but even low-end Ryzen Threadripper CPUs have TDP of 180W, which is a bit higher than I'd like. And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those, so one won't be able to return/replace a CPU if it doesn't work with ECC memory, AIUI, making it risky. Though I don't know how common i…

Keep in mind that Intel lies about its TDP.

They don’t. They just measure it differently than AMD. Intel measures at base clock, but AMD measures at sustained max clock IIRC. It’s definitely deceptive, but it’s not a lie as long as Intel tells you (which they do).

Re: ECC matters

#318

Post by everybodyknows »

cturner wrote: post #153
defanor wrote: post #66

Earlier quoted context omitted.

I've considered using an AMD CPU instead of Intel's Xeon on the primary desktop computer, but even low-end Ryzen Threadripper CPUs have TDP of 180W, which is a bit higher than I'd like. And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those, so one won't be able to return/replace a CPU if it doesn't work with ECC memory, AIUI, making it risky. Though I don't know how common i…

I went through this about a year ago, to build a low-TDP ECC workstation. I do not have stats on failure rates, just this anecdotal experience. Asrock and Asus seem to be the boards to get. For RAM, I got two sticks of Samsung M391A4G43MB1, and verified. The advice I remember from the forums was to stick to unbuffered ram (UDIMMS).

Did you consider any off-the-shelf ECC boxes?

Found some here -- bottom of the EPYC product line starts at $2849 ...!

https://www.velocitymicro.com/wizard.php?iid=337

Re: ECC matters

#319

Post by tjoff »

AaronFriel wrote: post #294
maria_weber23 wrote: post #274

Earlier quoted context omitted.

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

It can't eliminate it but: 1. Single bitflip correction along with Google's metrics could help them identify algorithms they've got, customer's VMs that are causing bitflips via rowhammer and machines which have errors regardless of workload 2. Double bitflip detection lets Google decide if they say, want to panic at that point and take the machine out of service, and they can report on what software was running or w…

ECC seems like a trivial thing to log and keep track of. Surely any Fortune 500 could do it and would have enough scale to get meaningful data out of it?

Re: ECC matters

#320

Post by colejohnson66 »

numlock86 wrote: post #201
marcosdumay wrote: post #72

Earlier quoted context omitted.

Keep in mind that Intel lies about its TDP.

Can you cite something? Sounds interesting.

It’s not true. Sortove. Intel measures at base clock while AMD does at sustained peak clock. Deceptive? Yes. Lie? No.
Post reply on HN