Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-ECC memory corrupted my hard drive image [video]

youtube.com

Post reply on HN
Print view View original

21–30 of 161 posts

Re: Non-ECC memory corrupted my hard drive image [video]

#21

Post by leguminous »

TacticalCoder wrote: post #2

> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's simply not that useful. Seriously: which percentage of developers have ECC on their development machine(s)? As developers we live in a world of SSH, cryptographic hashes, checksums everywhere, Git repositories (that is a big one), Merkle trees, digital signatures…

Checksums don't save you from memory corruption. If your data gets corrupted in memory, you will just end up checksumming and committing bad data. Or your checksum could get corrupted, and you commit a checksum that doesn't match your data. Checksums are more useful for safeguarding against disk or network corruption (although you shouldn't have network corruption issues over TLS or SSH).

Apparently Ryzen 7000 cpus can use ECC. I've heard reports that AMD needs to release an AGESA update, though, and ECC DDR5 memory availability is terrible. I'm hopeful that the situation will improve, because I also want to update my desktop. I've been using ECC memory since losing a filesystem on a desktop when a DIMM went bad.

Re: Non-ECC memory corrupted my hard drive image [video]

#22

Post by don-code »

H8crilA wrote: post #14

This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You can remedy it a bit (like with ECC), but ultimately there will always be corruption if you process hundreds of petabytes of data. Get used to it, your computer executes an instruction with a probability extremely close to 1, but not equal to 1. Deep in the ar…

Is this something that's documented publicly? I'd love to read more.

Re: Non-ECC memory corrupted my hard drive image [video]

#23

Post by dale_glass »

H8crilA wrote: post #14

This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You can remedy it a bit (like with ECC), but ultimately there will always be corruption if you process hundreds of petabytes of data. Get used to it, your computer executes an instruction with a probability extremely close to 1, but not equal to 1. Deep in the ar…

ECC isn't a terribly complicated technology, and can be used in all those cases.

In limited cases, a checksum is good enough. If you checksum outgoing data, and verify it on reception, then it being corrupted in transit whether on the network card or the cable can be detected and transparently compensated for.

Really, we can do much better than to "get used to it".

Re: Non-ECC memory corrupted my hard drive image [video]

#24

Post by H8crilA »

dale_glass wrote: post #23
H8crilA wrote: post #14

This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You can remedy it a bit (like with ECC), but ultimately there will always be corruption if you process hundreds of petabytes of data. Get used to it, your computer executes an instruction with a probability extremely close to 1, but not equal to 1. Deep in the ar…

ECC isn't a terribly complicated technology, and can be used in all those cases. In limited cases, a checksum is good enough. If you checksum outgoing data, and verify it on reception, then it being corrupted in transit whether on the network card or the cable can be detected and transparently compensated for. Really, we can do much better than to "get used to it".

You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :)

It can (and should! whenever possible) be improved, not fixed. There's always that pesky gamma that can hit a specific transistor, even if it is deep underground. Gamma cannot be fully stopped. At certain scales data corruption becomes directly measurable. And yes, corruption levels vary between pieces of hardware.

Re: Non-ECC memory corrupted my hard drive image [video]

#25

Post by jeroenhd »

It's good that with DDR5 consumer memory will get some super basic ECC on die, so hopefully the next generation of memory will make the problematic sticks more obvious (or prevent damage in the very least). ECC won't save you from memory corruption, but it'll save your data at least.

Personally, I would've just checksummed the individual failing files rather than the disk image and only back up the bad files separately. There are all kinds of ways for a disk image to fail and I wouldn't spend a second longer on it than absolutely necessary. The whole memtest permutation setup also would've been too much work for. E, I would just declare the motherboard faulty when two sticks that otherwise pass the test fail in specific configurations. A new motherboard is cheaper than super specific RAM sticks.

Re: Non-ECC memory corrupted my hard drive image [video]

#27

Post by moloch-hai »

willis936 wrote: post #20
moloch-hai wrote: post #13

Earlier quoted context omitted.

Because memtest only looks at values it wrote out very recently, before they have had a chance to flip. Memtest is looking for reliable failures, not evanescent one-off events.

SDRAM is continuously refreshing all cells. How long ago data was written doesn't make a big difference (aside from the case where you're reading data immediately after writing or reading that data).

This does not, of course, make any sense. Any given memory cell will be read once per, say, millisecond, and the value written back. Once it has flipped once, the wrong value will then be written back, and after that the wrong value is read back out and rewritten again, indefinitely. Errors are sticky, and accumulate. (Flipping back again is negligible unlikely.)

With ECC in the refresh path, such an error could be corrected and the right value would be overwritten over top of the bad one. Then errors would not accumulate, but would instead be "scrubbed". Mainframe machines scrub their RAM. Disks too.

Re: Non-ECC memory corrupted my hard drive image [video]

#28

Post by dale_glass »

H8crilA wrote: post #24
dale_glass wrote: post #23

Earlier quoted context omitted.

ECC isn't a terribly complicated technology, and can be used in all those cases. In limited cases, a checksum is good enough. If you checksum outgoing data, and verify it on reception, then it being corrupted in transit whether on the network card or the cable can be detected and transparently compensated for. Really, we can do much better than to "get used to it".

You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :) It can (and should! whenever possible) be improved, not fixed. There's always that pesky gamma that can hit a specific transistor, even if it is deep underground. Gamma cannot be fully stopped. At certain scales data corruption becomes directly meas…

Of course not. I'm not saying we can have perfection. I'm saying that we can do much better, using methods and technologies that are very old at this point.

The reason why we don't is laziness and market segmentation, mostly.

Re: Non-ECC memory corrupted my hard drive image [video]

#29

Post by gruez »

moloch-hai wrote: post #9

We don't have ECC mainly because Intel has long been hostile to "consumer" access to ECC. Apparently this was conceived as a market segmentation scheme: people outfitting servers could get ECC when they pay a huge premium. They would thereby not be tempted to cheap out and buy consumer-grade equipment, otherwise wholly adequate to meet all their needs at a radically cheaper price. That we cannot get laptops or even d…

> That we cannot get laptops or even desk machines with ECC, and so have them crash frequently, is seen as a trivial side effect of the strategy

I’m not sure what you mean by “frequently”, but my non-ECC machines definitely do not crash “frequently”.

> before MS, a program crashing was grounds for a refund

Source?

Re: Non-ECC memory corrupted my hard drive image [video]

#30

Post by newZWhoDis »

kevingadd wrote: post #15
TacticalCoder wrote: post #2

> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's simply not that useful. Seriously: which percentage of developers have ECC on their development machine(s)? As developers we live in a world of SSH, cryptographic hashes, checksums everywhere, Git repositories (that is a big one), Merkle trees, digital signatures…

I went out of my way for ECC because losing files can mean losing days of work. With how much a sweng gets paid it's worth the ECC premium if it saves me a few days of work over the life of the machine. Recently I discovered that one of my SSDs was quietly failing without setting off any warnings; doing a chkdsk showed that some files had already gotten corrupted. One of them was my backblaze backup index! Even thoug…

My favorite is when data silently corrupts, and then is happily propagated to your off-site recovery :/
Post reply on HN