My favorite example is a bit flip altering election results:
https://www.wnycstudios.org/podcasts/radiolab/articles/bit-f...
ECC matters
391–400 of 567 posts
Re: ECC matters
#392Earlier quoted context omitted.
> And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those ECC isn't validated by AMD for AM4 Ryzen models, but it's present and supported if the motherboard also supports it. Many motherboards have ECC support (the manual will say for sure), and a handful of models even explicitly advertise it as a feature. I have a Ryzen 9 3900X on an ASRock B450M Pro4 and 64 GB of ECC DRAM,…
What do you mean by “validated”? There’s the silicon, but they don’t test it?
IMO, "validated" is intentionally wishy-washy and mostly means that AMD would prefer it if enterprises paid them more money by buying EPYC (or Ryzen Pro) parts instead of consumer Ryzen parts. Much like how Intel prefers selling higher-margin Xeons over Core i5. It's market segmentation, but friendlier to consumers than Intel's approach.
Re: ECC matters
#393I still remember Craig Silverstein being asked what his biggest mistake at Google was and him answering "Not pushing for ECC memory." Google's initial strategy (c. 2000) around this was to save a few bucks on hardware, get non-ECC memory, and then compensate for it in software. It turns out this is a terrible idea, because if you can't count on memory being robust against cosmic rays, you also can't count on the soft…
Figure this is as good of a time as any to ask this: There are many various DRAMs in a server (say, for disk cache). Has Google or anyone who operates at a similar scale seen single bit errors in these components?
This is as old as computing and predates Google.
When America Online was buying EV6 servers as fast as DEC could produce them, they used to see about about 1 double bit error per day across their server farm that would reboot the whole machine.
DRAM has only gotten worse--not better.
Re: ECC matters
#394Earlier quoted context omitted.
I've considered using an AMD CPU instead of Intel's Xeon on the primary desktop computer, but even low-end Ryzen Threadripper CPUs have TDP of 180W, which is a bit higher than I'd like. And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those, so one won't be able to return/replace a CPU if it doesn't work with ECC memory, AIUI, making it risky. Though I don't know how common i…
I went through this about a year ago, to build a low-TDP ECC workstation. I do not have stats on failure rates, just this anecdotal experience. Asrock and Asus seem to be the boards to get. For RAM, I got two sticks of Samsung M391A4G43MB1, and verified. The advice I remember from the forums was to stick to unbuffered ram (UDIMMS).
Yes, the consumer parts only support UDIMMs. If you want RDIMMs, you have to pay for EPYC.
Re: ECC matters
#395Earlier quoted context omitted.
You could run memtest on a pc without ecc for a couple of days and to estimate the error rate, or not?
If you manage to replicate bitflips every few days your RAM is broken. It's the "once every other year" type of bitflip that's the problem. The proverbial "cosmic ray" hitting your DRAM and flipping a bit. That will be caught by ECC but it'll most likely remain a total mystery if it causes your non-ECC hardware to crash.
It isn't only cosmic rays. Regular old radiation can also cause it. I've read about a server that had many repeated problems and the techs replaced the entire motherboard at one point.
Then one of them brought in his personal Geiger counter and found the radiation coming off the steel in that rack case was significantly higher than background.
You may never know when the metal you use was recycled from something used to hold radioactive materials.
Re: ECC matters
#396Earlier quoted context omitted.
Bit flips can happen, but regardless if they can get repaired by ECC code or not, the OS is notified, iirc. It will signal a corruption to the process that is mapped to the faulty address. I suppose that if the memory contains code, the process is killed (if ECC correction failed).
> I suppose that if the memory contains code, the process is killed (if ECC correction failed). Generally, it would make the most sense to kill the process if the corrupted page is data , but if it's code, then maybe re-load that page from the executable file on non-volatile storage. (You might also be able to rescue some data pages from swap space this way.)
If you go that route, you should be able to avoid the code/data distinction entirely; as data pages can also be completly backed by files. I believe the kernel already keeps track of what pages are a clean copy of data from the filesystem, so I would think it would be a simple matter of essentially pageing out the corrupted data.
What would be interesting is if userspace could mark a region of memory as recomputable. If the kernel is notified of memory corruption there, it triggers a handler in the userspace process to rebuild the data. Granted, given the current state of hardware; I can't imagine that is anywhere near worth the effort to implement.
Re: ECC matters
#397Earlier quoted context omitted.
But it isn't just a crash, it's also silent data corruption that will never be detected
It can be a concern, that's true, but personally most of the stuff I edit end up checked into a git repository or something similar. And I mean, we all spend all day editing test messages and comments and files on non-ECC hardware, yet bitflip-induced corruption is rare enough that I can't say that I've witnessed a single instance of it in my life, despite spending a good chunk of it looking at screens. It's just not…
Can make that statement with any certainty? My personal and family computers have crashed quite a few times, and have corrupted photoes and files, some of them are valuable (taxes, healthcare, etc. Personal computers have valuable data these days)
I couldn't tell, as a user, which if those corruptions and crashes were causes by bitflips. Could you?
Re: ECC matters
#398You don't need to look at kernel crashes to speculate about bus and memory errors -- just check the logs on a few systems that do have ecc. Pretty soon you'll see correctable errors being reported.
I don’t know much about this topic, but is it possible that ECC memory is more prone to single bit errors than non-ECC memory because there is less pressure on companies to minimize such errors? If this were the case, it would skew the data.
Re: ECC matters
#399Earlier quoted context omitted.
It’s not. I have a low end Epyc machine with ECC. It has a TDP of something like 30 watts.
I didn't consider embedded CPUs (I guess that's about an embedded EPYC, not a server one), those look neat. But there's no official ECC support (i.e., it's similar to Ryzen CPUs), is there? Edit: as detaro mentioned in the reply, there is, and here's the source [0] -- that's what they mean by "RAS" on promotional pages [1]. That indeed looks like a nice option. [0] https://www.amd.com/system/files/documents/updated-3…
RAS covers more than just DRAM, but yes. Historically, the reporting interface is called MCA (Machine Check Architecture) / MCE. I think both AMD and Intel have extensions with other names, but MCA/MCE points you in the right direction.
Re: ECC matters
#400Earlier quoted context omitted.
> it's not true that ECC is supported well on AMD ECC is supported on most Ryzen models[1], as long as the motherboard supports it. In fact, ASUS and ASRock (possibly others) have Ryzen motherboards designed for workstation/server use where ECC support is specifically advertised. [1] The only exception is the Ryzen CPUs with integrated graphics.
Depends what you mean by supported. Semi-offically: ECC is not disabled. It works, but not validated for our consumer client platform. Validated means run it through server/workstation grade testing. For the first Ryzen processors, focused on the prosumer / gaming market, this feature is enabled and working but not validated by AMD. You should not have issues creating a whitebox homelab or NAS with ECC memory enabled…
Your quote is for consumer platforms (Ryzen) only; GP's statement was that ECC is not well-supported on AMD at all, which is obviously false (EPYC, Threadripper).