Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ECC and DDR5

etbe.coker.com.au

Post reply on HN
Print view View original

101–110 of 162 posts

Re: ECC and DDR5

#101

Post by hangonhn »

"I have used more than a few systems with ECC UDIMMs and I recommend avoiding them if convenient as ECC UDIMMs are expensive on the second hand market while ECC RDIMMs can get very cheap."

Boy did I learn that the hard way. I found an ASRock server motherboard that fit my needs for a NAS perfectly. I just assumed all ECC memory were registered so I didn't even bother to check. I had already put everything together minus the memory. I had actually purchased the memory and had to return them because they were RDIMMs and get the ECC UDIMMs for quite a bit more. Luckily this was just before the AI induced price spike.

Re: ECC and DDR5

#102

Post by tbrownaw »

zargon wrote: post #36
tbrownaw wrote: post #27

Earlier quoted context omitted.

> There is no world where accepting data corruption in order to save 3% cost is a reasonable tradeoff. It is if you're doing something where corruption is detectable after the fact, and happens rarely enough that redoing affected work adds less than 3% overhead.

If you don't have ECC, you don't even know that your CPU is executing the correct code. There goes the "detectable" part.

There's a decent amount of distributed stuff that has a design principle of not trusting the hardware it's running on.

Re: ECC and DDR5

#103

Post by throwaway270925 »

howard941 wrote: post #91

ECC is essentially mandatory for ZFS. Assuming that is you prefer your data uncorrupted.

> ECC is essentially mandatory for ZFS

Please stop repeating this old myth, it is - and always has been - BS and needs to die.

See for example: https://kldload.com/zfs-wiki/myths

Re: ECC and DDR5

#104

Post by etbe »

flaunf221 wrote: post #97
piguin wrote: post #94

Earlier quoted context omitted.

You need the ECC RAM for your filesystem with integrity only then can you be confident you aren't going to be familiar with a concept like this happening 9 or 10 times.

You can read online how often correctable errors happen to people with ECC and reporting. Single digit - sometimes none - errors per year in systems with consumer amount of RAM.

Systems with "consumer amounts of RAM" are often run with what you might call "consumer amounts of sysadmin knowledge" and not notice the errors.

One time I took a working workstation from my home to my parents' home to give them. When I booted it up it gave a lot of strange kernel errors which I googled and discovered were reports of ECC correction. I reseated all DIMMs and it was fine. If that gave errors a few times a year instead of thousands of times on the first boot I wouldn't have noticed.

Re: ECC and DDR5

#105

Post by etbe »

creatonez wrote: post #95
flaunf221 wrote: post #93

Earlier quoted context omitted.

> You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512. And 9 times out of 10 (probably much more) it will be faulty storage, not RAM. I'd say you are way overestimating impact of no-ECC.

...No? Faulty storage will almost always show up as I/O errors, modern storage generally doesn't want to send a block of data to the CPU if there is an error in it. If your bits are flipping, chances are it's RAM and not storage. Your storage already has ECC. Some crappy flash drives might not do this, though. Seems to behave like a floppy disk at times.

The majority of storage errors I've seen since using BTRFS and ZFS everywhere have shown up only as the storage returning corrupt data and claiming it to be good. I've seen drives and SSDs return dozens of errors per month for months on end without any SAS or SATA errors or any SMART errors.

I've seen this in DC grade SSDs, NAS grade hard drives, and SSDs sold in high end laptops.

Re: ECC and DDR5

#106

Post by [deleted] »

throwaway270925 wrote: post #103
howard941 wrote: post #91

ECC is essentially mandatory for ZFS. Assuming that is you prefer your data uncorrupted.

> ECC is essentially mandatory for ZFS Please stop repeating this old myth, it is - and always has been - BS and needs to die. See for example: https://kldload.com/zfs-wiki/myths

[deleted]

Re: ECC and DDR5

#107

Post by etbe »

adrian_b wrote: post #45
inigyou wrote: post #22

Earlier quoted context omitted.

My Linux installation came with an AMD MCE driver that reports them to dmesg. It logs a few corrected errors a day.

A few errors per day seems much too frequent, unless you live at a high altitude. Normal good DIMMs, at least when new, should not have errors more frequently than one error per many months. Frequent errors may appear with memories that are not seated well in their sockets, or which are old, at least several years old. Frequent errors may also be caused by more general computer problems, like a bad power supply unit.

Does this imply that the all plastic gamer PCs (that are more designed to be art projects than serious computers) are bad and that everyone should be using systems with solid steel cases to block radiation?

Re: ECC and DDR5

#108

Post by etbe »

adrian_b wrote: post #86
willis936 wrote: post #76

Earlier quoted context omitted.

That's a much weaker statement than "the primary source of uncorrelated bit flips is cosmic radiation".

Cosmic radiation is always a source of errors for any kind of DRAM. Whether it is the primary source of errors, depends on the design of the DRAM die. In a badly designed DRAM there could be many other error causes. The designers of a DRAM certainly attempt to minimize all the error causes that they can control. If they succeed, the cosmic radiation would remain the primary error source. Without access to internal do…

Companies making DRAM have no incentive to make the best products theoretically possible, they just want them good enough to not get many returns and avoid the risk of a class action lawsuit if a significant pattern of problems appears.

Avoiding returns for ECC DIMMs is a harder challenge when they are used in an environment where the server will have a management system that tracks errors over all time and when there might be a network management system tracking all errors over all systems.

Avoiding returns for non-ECC DIMMs is easy when a few errors per month won't be noticed by most users who assume that Windows always crashes anyway.

Has any DRAM company tested chips and made non-ECC DIMMs from the ones that have lower quality?

As an aside are the 2^N*3 sizes of DDR5 DIMMs (like 24G and 48G) made from chips that had errors in one section and got reconfigured to have 3/4 the capacity to not use the bad parts?

Re: ECC and DDR5

#109

Post by bartleeanderson »

ck2 wrote: post #7

16GB ECC DDR5 was $50 early 2025 it is now $500 when the AI/datacenter bubble finally bursts we are going to have some amazing hardware deals

I had a conversation with chatgpt about that very thing, whether it was a bubble or not and how the big players could absorb the cost and where it would all land and the interesting thing was if China gets its foothold and grabs enough share because the prices are too high even being subsidized, that could cause real trouble. But the point was its not a bubble in the conventional sense. Have your own conversation about that with an AI.

Re: ECC and DDR5

#110

Post by TacticalCoder »

nu11ptr wrote: post #51

I've always thought it odd that we don't use ECC as a standard on all computers. I think people really downplay the impact of memory issues. They can be devastating, especially over time. They will slowly corrupt your file system, documents, binary files, code, everything. You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512. You are then left with t…

> I think people really downplay the impact of memory issues.

I don't think so. I've got my important data backed up on offline HDDs and SSDs and my most important data lives on a RAID system (ZFS) on a Xeon with ECC RAM but...

... there are simply a huge lot of checks everywhere making it so that bit flips on non ECC systems simply aren't that serious.

We live in a world of checksums and retries at every layer and distributed systems that are just about everywhere.

I went on for decades without ECC memory and nothing bad happened. Yes, I had the occasional bitflips. I did found out faulty mem using memtest (or weirder stuff, like some RAM stick identical to the other three, same chip everywhere, same sticker: but reported by memstick as being some other mem).

But I don't think that the "one bitflip cascading into corruption of your life's data" is something necessary to freak about.

It's not happening. Never happened to anyone.

If you (or other) are really concerned, you can use my scheme:

    - dsc09874.jpg
becomes:

    - dsc09874-b3-73b0835221.jpg
which means "that file's Blake3 checksum begins with 73b0835221" (use something else than Blake3 if so desired).

About 1/3rd of all my files are "tagged" this way, directly in the filename: this survive different medium and different OSes.

I then have my custom rsync backuping script doing a dry-run and warning me in a "stop the world!" fashion if it detects a bogus checksum (either on the source or on the dest: during the dry-run any modification to a file with a checksum is verified and any checksum mismatch means "stop the world").

I then also have, in addition to my custom backuping script, another script that allows me to verify either all files with that naming convention or a percentage I pick of those files (so basically some random sampling).

It's not mutually exclusive with ZFS / a system with ECC RAM.

> but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time

I hear you but... How much do you really need, years later, a file that you haven't needed for years? That one file is unlikely to be a big loss.

Now I'm not saying you're wrong: I ended up with my naming scheme (appending a partial cryptographic hash to the filename) after finding a series of about 20 pictures that were corrupted. And on all the backups I first checked, they were also corrupted. I ended up finding a very old data CD on which that series of .jpg files were fine. And I decided "never again, I want such SNAFU to be detected in the future".

But truth is: with 40 000 pictures, life without those 20 pictures would still be fine.

Post reply on HN