Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-ECC memory corrupted my hard drive image [video]

youtube.com

Post reply on HN
Print view View original

141–150 of 161 posts

Re: Non-ECC memory corrupted my hard drive image [video]

#141

Post by Too »

nirvdrum wrote: post #118
eek2121 wrote: post #112

Earlier quoted context omitted.

I see a ton of replies here. I am a tech user with thousands of terabytes stored locally. Most of those terabytes are mirrored in the cloud. In all of those situations I've never actually had any corruption related to memory or storage. I have 932gb in PICTURES ffs. I get the claimed need for EEC, but yet I don't. I own a number of web properties and regularly update logos, software packages, html files, PHP files, r…

How do you confirm that there's no corruption? My understanding was a silent bit flip would be difficult to detect. I ended up using ZFS and ECC memory to avoid silent corruption, but I'm largely relying on the motherboard to do the right thing when an ECC error is detected. It'd be nice to verify after the fact.

git fsck

Re: Non-ECC memory corrupted my hard drive image [video]

#142

Post by exikyut »

mrlonglong wrote: post #90

We've got a server that keeps rebooting due to a bad ECC DIMM chip. I thought the whole point of ECC was to keep the server going until we can replace the DIMM?

IIUC it's the kernel EDAC subsystem rebooting the server for you. If you turn it off you'll get non-ECC behaviour, but in this case you'll also get really interesting system issues because as the sibling comments have said your RAM is actually faulty :P

Re: Non-ECC memory corrupted my hard drive image [video]

#143

Post by justinclift »

Near the end, the video mentions his (old) computer "apparently can't take 'high density' memory".

Then the author goes through a process of randomly buying memory and hoping it works.

Anyone know what that "high density" memory problem is about? Maybe it's a misunderstanding of memory channels and ranks?

Re: Non-ECC memory corrupted my hard drive image [video]

#144

Post by Gordonjcp »

PragmaticPulp wrote: post #103
Filligree wrote: post #95

Earlier quoted context omitted.

You can't, that's the thing, right? I have a Ryzen desktop with ECC, and it registers about one bit-flip per week. I don't know how many of those would become crashes, but I'm more worried about the ones that wouldn't.

> I have a Ryzen desktop with ECC, and it registers about one bit-flip per week. This isn’t normal. You have a bad memory module. A non-ECC machine should able to support memtest86 (a rigorous memory testing tool) for a week straight without a single bitflip. Having a bit flip per week is so far away from normal that it’s definitely a bad memory part.

I'd hate to think what all those neutrinos would be doing to you. You are very much larger than a memory cell, for every bit flip you must be catching a trillion neutrinos.

Maybe OP's house is in the Van Allen Belt or something?

Re: Non-ECC memory corrupted my hard drive image [video]

#145

Post by viraptor »

szundi wrote: post #140
eek2121 wrote: post #112

Earlier quoted context omitted.

I see a ton of replies here. I am a tech user with thousands of terabytes stored locally. Most of those terabytes are mirrored in the cloud. In all of those situations I've never actually had any corruption related to memory or storage. I have 932gb in PICTURES ffs. I get the claimed need for EEC, but yet I don't. I own a number of web properties and regularly update logos, software packages, html files, PHP files, r…

You can be pretty sure that you have some corrupt bits

No, we really can't be sure. I'm annoyed when people are sure about it without posting their ECC correction rate stats from live systems.

I'm continuously writing metrics on a non-ECC system. There are 3 places that the bitflip can affect the data: pre-writing (checksum will be correct), when flushing to the drive (checksum mismatch), calculating the checksum (mismatch again). There may be some silent corruption (in 1 out of 3 cases), but I've been scrubbing the drive every day for ages and there's not a single error.

Non-ECC seems to be doing well. Other claims could really use actual numbers.

Re: Non-ECC memory corrupted my hard drive image [video]

#146

Post by TomSwirly »

H8crilA wrote: post #14

This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You can remedy it a bit (like with ECC), but ultimately there will always be corruption if you process hundreds of petabytes of data. Get used to it, your computer executes an instruction with a probability extremely close to 1, but not equal to 1. Deep in the ar…

> ultimately there will always be corruption if you process hundreds of petabytes of data.

Your statement is in practical terms false.

Error correction exists, and adding enough redundancy to make any corruption have such a tiny chance of being undetected that the whole history of the universe could go by without it happening is a problem that was solved over 60 years ago with the invention of the Reed-Solomon code,

Re: Non-ECC memory corrupted my hard drive image [video]

#147

Post by pedrocr »

encryptluks2 wrote: post #33

I watched the full video. It was long but very informative. The humor at times made up for the length and the presenter showed a lot of deep knowledge that most people won't have. My biggest gripe is that they just didn't try replacing the RAM sticks in the first place. I get that they wanted to do a root cause analysis, but geez the time and patience they had to do all those memory tests. No wonder they did a video…

There's no adequately priced ECC UDIMMs because the market is so small. Servers use registered so it's a very small niche. But for the gaming overclock application it would actually be a great fit. Because ECC gives you a great early indication of failure, verifying that an overclock is stable enough to use becomes much easier. My hope for ECC availability is that one of the gaming brands decides to upsell it as a feature and then we can build nice workstations with fast ECC RAM. I'll even put up with RGB for it if I must. AMD board manufacturers have been in a great position to do this since the beginning of Ryzen. Too bad none of them have.

Re: Non-ECC memory corrupted my hard drive image [video]

#148

Post by to11mtm »

H8crilA wrote: post #24
dale_glass wrote: post #23

Earlier quoted context omitted.

ECC isn't a terribly complicated technology, and can be used in all those cases. In limited cases, a checksum is good enough. If you checksum outgoing data, and verify it on reception, then it being corrupted in transit whether on the network card or the cable can be detected and transparently compensated for. Really, we can do much better than to "get used to it".

You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :) It can (and should! whenever possible) be improved, not fixed. There's always that pesky gamma that can hit a specific transistor, even if it is deep underground. Gamma cannot be fully stopped. At certain scales data corruption becomes directly meas…

> You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :)

I thought L2/L3 is ECC (at least on Intel, though L1 I think is parity only)

Re: Non-ECC memory corrupted my hard drive image [video]

#149

Post by to11mtm »

justinclift wrote: post #143

Near the end, the video mentions his (old) computer "apparently can't take 'high density' memory". Then the author goes through a process of randomly buying memory and hoping it works. Anyone know what that "high density" memory problem is about? Maybe it's a misunderstanding of memory channels and ranks?

Not sure if it's what he means, but in my experience 'High density' refers to less chips for a given capacity.

When I used to sling PC hardware, as memory densities increased, you would run into things like 'This Motherboard will only take a 256MB module if it has 8 chips on both sides (16 chips total), it will not take a 256MB module with 8 chips on one side (8 chips total)' Depending on the board, it might only register part of the capacity and be stable, might register part of the capacity and be unstable, or just not boot at all.

Re: Non-ECC memory corrupted my hard drive image [video]

#150

Post by to11mtm »

mrlonglong wrote: post #90

We've got a server that keeps rebooting due to a bad ECC DIMM chip. I thought the whole point of ECC was to keep the server going until we can replace the DIMM?

ECC Corrects 1 bit errors and detects 2 bit errors. It does not handle all hardware failures however.

An actual bad chip may fall into a different category, I think that's when technologies like Chipkill[0] might come into play.

[0] - https://en.wikipedia.org/wiki/Chipkill

Post reply on HN