I've always thought it odd that we don't use ECC as a standard on all computers. I think people really downplay the impact of memory issues. They can be devastating, especially over time. They will slowly corrupt your file system, documents, binary files, code, everything. You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512. You are then left with this "icky" feeling. What parts of your work are compromised? Obviously this is far worse with things like binary files and structures than those built on readable text. I have normally just rebuilt my system from scratch in these instances (after fixing the issue of course), but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time. You might not find them until years later.
ECC and DDR5
51–60 of 162 posts
Re: ECC and DDR5
#52Got ECC udimm for my ddr4 server and was surprised to see it picking up errors occasionally (once every couple months). Likely from a weakness in one of the sticks. This far it’s always corrected it though so opted to keep them anyway (nobody wants to be minus 32gb in these trying memory times) With normal sticks I’d not have know that there is a potential issue.
Depending on your server's CPU sockets, reseating the CPU might help if they were ever swapped before (esp if you got the server used). I have a marginally bad RAM slot on one of my Supermicro X11 boards because my dumb ass dropped the CPU into the socket and bent a few pins. I bent them back, but you can only do so much and the furthest RAM slot won't work without throwing errors daily if I run it at 2400MHz.
Re: ECC and DDR5
#53Something important not discussed here is that you need not only error correction (when possible) but also error reporting, so you can know you RAM is failing. Sadly, is not possible to get DDR5 on-die ECC error counts reported to the OS. Alternative solution, use IBECC with DDR5, where the CPU reserves a portion of the RAM for parity bits and does the check / reporting.
My Linux installation came with an AMD MCE driver that reports them to dmesg. It logs a few corrected errors a day.
Check if those are all coming from DIMMs, as opposed to other subsystems (on-die caches).
Re: ECC and DDR5
#54Interesting article, and I agree that ECC needs to be more widely available. But I think this is a major overreach: > We need ECC RAM to be more widely used. Ideally we would have some government action to force this given the ongoing cost to society in corrupted data and lost time due to RAM hardware errors. I think that at minimum we need sufficient taxes on non-ECC RAM (and EC4 RAM for DDR5) to make it more expens…
> ECC RAM is not needed for every application and mandating it for every gaming rig and machine used to run Chrome will just increase prices unnecessarily. The price increase will be very small if manufacturers are no longer able to segment the market to make a higher margin on those who need it. No one wants even games or web browsers to crash randomly. > If they want to eat that cost, it should be up to them. They…
>The price increase will be very small if manufacturers are no longer able to segment the market to make a higher margin on those who need it.
AMD doesn't have this segmentation. Where's the hoard of people building systems with ECC ram? At best you have a few people on HN saying they're doing it, but in practice nobody actually does it.
Re: ECC and DDR5
#55This is not a very good post.
Claiming that every issue you've experienced is from seating errors? Come on man, at least deliberately reproduce that scenario once. I've tortured motherboards for fun and never managed that on purpose. We did pry every cap off of an old K6 board while it was on and... it just kept running. It didn't come back after we turned it off
The amount of time it takes to reseat memory, even if the case is already open and ready, is also more than enough time for it to cool back down.
If you have decent memory running at what it's rated for and you're seeing more than a couple per year, good odds are it's getting too hot or the load line calibration is not aggressive enough.
E: I have never, and I mean literally never, seen age or abuse play a factor.
Re: ECC and DDR5
#56Earlier quoted context omitted.
Curious to learn more here - why would altitude play a role here? And when you say high altitude - are we talking La Paz, Bolivia (~12k feet) or Denver, CO (~5k feet)?
The primary source of uncorrelated bit-flips is, effectively, cosmic radiation. (Correlated bit-flips are likely from a manufacturing error... or, at least in one case, from excessively radioactive ceramic packaging.) The more atmosphere you have to randomly absorb high-energy photons, the better. Higher density memory drops fewer electrons in each well, which means a lower-energy photon can change the state. Corolar…
>The primary source of uncorrelated bit-flips is, effectively, cosmic radiation.
This gets said a lot but with little evidence. The mario speedrun bit flip has been replicated with marginal connector insertion.
Re: ECC and DDR5
#57Earlier quoted context omitted.
AMD, Intel, and motherboard manufacturers do qualify the CPUs and motherboards for "overclocked" (above JEDEC) memory speeds. The systems are designed to work at the overclocked transmission speeds. They need to be paired with memory modules that are capable of functioning at those speeds. With ECC presently being a datacenter thing, companies only make it according to JEDEC specs. If ECC were standard for everyone,…
The motherboard manufacturers test their motherboards, to ensure that they typically work with overclocked memory modules. If you buy a complete gaming computer which includes overclocked memory, then hopefully the vendor has done a burn-in and has tested for some time the computer, as sold. However, if you buy the CPU yourself, neither Intel nor AMD provides any guarantee that the CPU will work with overclocked memo…
> However, if you buy the CPU yourself, neither Intel nor AMD provides any guarantee that the CPU will work with overclocked memory.
There's no guarantee that it works with standard clocked memory either.
Re: ECC and DDR5
#58The problem I have with ECC discussions is that it’s always anecdotal. There is one Google study finding memory flips with some probability in their datacenters, but it was awhile ago. Everyone repeats the statement DDR5 on-die ECC isn’t a replacement for regular ECC, but nobody ever shows data on DDR5 bitflips. That’s not a knock on this article per se, but it’d be nice to have actual data here.
i have years and terabytes of ECC DDR4 and DDR5 memories, UDIMM and RDIMM, with edac enabled all this time, and i have literally never once observed a single error correction. however, (1) i built all these computers myself (2) the servers are located on the middle floor a thick, concrete multi-story warehouse-to-office conversion. as others have said, on a correctly built computer, the source of bit flips is cosmic radiation, and there is a lot of mass between these machines and the sky.
Re: ECC and DDR5
#59Earlier quoted context omitted.
AMD, Intel, and motherboard manufacturers do qualify the CPUs and motherboards for "overclocked" (above JEDEC) memory speeds. The systems are designed to work at the overclocked transmission speeds. They need to be paired with memory modules that are capable of functioning at those speeds. With ECC presently being a datacenter thing, companies only make it according to JEDEC specs. If ECC were standard for everyone,…
The motherboard manufacturers test their motherboards, to ensure that they typically work with overclocked memory modules. If you buy a complete gaming computer which includes overclocked memory, then hopefully the vendor has done a burn-in and has tested for some time the computer, as sold. However, if you buy the CPU yourself, neither Intel nor AMD provides any guarantee that the CPU will work with overclocked memo…
The ECC on the GTX 1080 let me figure out the optimal overclock with the only feedback mechanism being benchmarked throughout.
Re: ECC and DDR5
#60> We need ECC RAM to be more widely used. Ideally we would have some government action to force this given the ongoing cost.... The rest of this article is great. This, however, is absolute garbage and nonsense. The last thing you want is more regulations and "government certified systems". Take a look at the bullshit with age verification bills and operating systems.
"This, however, is absolute garbage and nonsense. The last thing you want is more regulations and government certified systems." — the guy who designed the Ford Pinto