Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC matters

realworldtech.com

Post reply on HN
Print view View original

291–300 of 567 posts

Re: ECC matters

#291

Post by slumdev »

maria_weber23 wrote: post #274
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

Error-correcting code (the "ECC" in ECC) is just a quorum at the bit level.

Re: ECC matters

#292

Post by mongol »

mlyle wrote: post #250
ethanpil wrote: post #236

Earlier quoted context omitted.

Built a NAS. My #1 concern for choosing CPU was TDP. This machine is on 24/7 and power use is a primary concern where I live because electricity is NOT cheap.

This is a poor way to make the choice. TDP is supposed to specify the highest power you can get the processor to dissipate, not typical or idle use. And since different manufacturers specify TDP differently, you can't even compare the number. Performance/watt metrics and idle consumption would have been a far better way to make this choice. If you have a choice between A) something that can dissipate 65W peak for 100…

Is there a metric to look for to understand what power consumption is at "idle" or something close to that? That is what confuses me. I don't want to spend a lot of money on something that will be always on, and usually idling, and finding that its power usage is way higher than I thought. But perhaps there is a metric that tells that. I have not looked closely at it.

Also, even though the CPU may draw less, can still the power supply waste more, just because it is beefy? Comparing with a sports car, they have great performance, but also use more gas in ordinary traffic? Can a computer be compared with that?

Re: ECC matters

#293

Post by hn3333 »

maria_weber23 wrote: post #274
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

Bit flips can happen, but regardless if they can get repaired by ECC code or not, the OS is notified, iirc. It will signal a corruption to the process that is mapped to the faulty address. I suppose that if the memory contains code, the process is killed (if ECC correction failed).

Re: ECC matters

#294

Post by AaronFriel »

maria_weber23 wrote: post #274
nostrademons wrote: post #237

I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…

ECC memory can't eliminate the chances of these failures entirely. They can still happen. Making software resilient against bitflips in memory seems very difficult though, since it not only affects data, but also code. So in theory the behavior of software under random bit flips is well... Random. You probably would have to use multiple computers doing the same calculation and then take the answer from the quorum. I…

It can't eliminate it but:

1. Single bitflip correction along with Google's metrics could help them identify algorithms they've got, customer's VMs that are causing bitflips via rowhammer and machines which have errors regardless of workload

2. Double bitflip detection lets Google decide if they say, want to panic at that point and take the machine out of service, and they can report on what software was running or why. Their SREs are world-class and may be able to deduce if this was a fluke (orders of magnitude less likely than a single bit flip), if a workload caused it, or if hardware caused it.

The advantage the 3 major cloud providers have is scale. If a Fortune 500 were running their own datacenters, how likely would it be that they have the same level of visibility into their workloads, the quality of SREs to diagnose, and the sheer statistical power of scale?

I sincerely hope Google is not simply silencing bitflip corrections and detections. That would be a profound waste.

Re: ECC matters

#295

Post by rajesh-s »

type0 wrote: post #54

Consumer awareness about ECC needs to be better, with recent security implications I simply can't understand why more motherboard manufacturers don't support it on AMD. Intel of course is all to blame on the blue side, I stopped buying their overpriced Xeons because of this.

Good point on the need for awareness!

The industry has convinced the average user of consumer hardware that PPA (Power,Performance,Area) is all that needs to get better with generational improvements. Hoping that the concerning aspects of security and reliability that have come to light in the recent past changes this.

Re: ECC matters

#296

Post by Cloudef »

mlyle wrote: post #271
mongol wrote: post #255

Earlier quoted context omitted.

How do you do that? Is it a setting in the bios? Or can it be done runtime? If so, how? It sounds interesting if I can run a beefy rig as a power efficient device, for always-on scenarios, and then boost it when I need.

> How do you do that? Is it a setting in the bios? Or can it be done runtime? On AMD, it's a utility you run. I believe you may require a reboot to apply it. On some Intel platforms, it's been settings in the BIOS. > It sounds interesting if I can run a beefy rig as a power efficient device, for always-on scenarios, and then boost it when I need. This is what the processor is doing internally anyways. It throttles vo…

Ryzen Master doesnt seem to be available for linux so you end up with bunch of unnofficial hacks that may or may not work. I run sff setup myself, originally wanted to get 3600 but it was out of stock, and the next tdp friendly processor was 3700x.

Re: ECC matters

#297

Post by adrian_b »

defanor wrote: post #128
usefulcat wrote: post #114

Earlier quoted context omitted.

It’s not. I have a low end Epyc machine with ECC. It has a TDP of something like 30 watts.

I didn't consider embedded CPUs (I guess that's about an embedded EPYC, not a server one), those look neat. But there's no official ECC support (i.e., it's similar to Ryzen CPUs), is there? Edit: as detaro mentioned in the reply, there is, and here's the source [0] -- that's what they mean by "RAS" on promotional pages [1]. That indeed looks like a nice option. [0] https://www.amd.com/system/files/documents/updated-3…

For embedded applications, there is official ECC support for all CPUs named Epyc or Ryzen Vxxxx or Ryzen Rxxxx.

There are computers in the Intel NUC form factor, with ECC support (e.g. with Ryzen V2718), e.g from ASRock Industrial.

Re: ECC matters

#298

Post by simias »

andi999 wrote: post #102
simias wrote: post #26

Earlier quoted context omitted.

The problem is that, if you don't have ECC to detect the errors, it's very hard to know what exactly caused a random, non-reproducible crash. Especially in kernel mode where there's little memory protection and basically any driver could be writing anywhere at any time. I can understand Linus's frustration from that point of view: without ECC RAM when you get some super weird crash report where some pointer got corru…

You could run memtest on a pc without ecc for a couple of days and to estimate the error rate, or not?

If you manage to replicate bitflips every few days your RAM is broken.

It's the "once every other year" type of bitflip that's the problem. The proverbial "cosmic ray" hitting your DRAM and flipping a bit. That will be caught by ECC but it'll most likely remain a total mystery if it causes your non-ECC hardware to crash.

Re: ECC matters

#300

Post by alkonaut »

louwrentius wrote: post #275
alkonaut wrote: post #110

Earlier quoted context omitted.

I know what it does, but I still don’t care (so long as it costs money or even 1% performance). It’s a tradeoff between money/performance and the frequency of crashes, corruption etc. Bit rot is just one of many threats to my data. Backups take care of that as well as other threats like theft, fire, accidental deletion. This is similar to my reasoning around the recent side channel attacks on intel CPUs. If I had a c…

> Bit rot is just one of many threats to my data. Backups take care of that as well as other threats like theft, fire, accidental deletion. No, that's the big mistake people make: backups just backup bit-rotted data, until it is too late and the last good version is rotated out and lost forever.

I’m aware. But the risk is extremely small (and 99.9% of important data is not created on the machine but goes directly from e.g iOS camera to backup).

My desktop machine is basically a gaming rig with disposable data. Hence the “performance over integrity”.

I also never rotate anything out. Every version of everything is in the backups. Storage is that cheap these days.

Post reply on HN