Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-ECC memory corrupted my hard drive image [video]

youtube.com

Post reply on HN
Print view View original

11–20 of 161 posts

Re: Non-ECC memory corrupted my hard drive image [video]

#11

Post by ubercow13 »

tasubotadas wrote: post #7
TacticalCoder wrote: post #2

> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's simply not that useful. Seriously: which percentage of developers have ECC on their development machine(s)? As developers we live in a world of SSH, cryptographic hashes, checksums everywhere, Git repositories (that is a big one), Merkle trees, digital signatures…

When people complain about these random OS crashes and freezes it's usually RAM corruption at fault. Per 1gb of RAM, you can expect to see 266 bit errors per month[1-2] if you are using your PC 16h per day. Multiply that by 64GB or 128GB of RAM and it's crazy to think that you won't run into any of the stability issues. [1-2] https://static.googleusercontent.com/media/research.google.c... [1-2] https://en.wikipedia.o…

>266 bit errors per month

That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?

Re: Non-ECC memory corrupted my hard drive image [video]

#12

Post by dale_glass »

TacticalCoder wrote: post #2

> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's simply not that useful. Seriously: which percentage of developers have ECC on their development machine(s)? As developers we live in a world of SSH, cryptographic hashes, checksums everywhere, Git repositories (that is a big one), Merkle trees, digital signatures…

> But really most of all I live in a world of Git repositories. My entire Emacs config is versioned under Git (I know YMMV but I like it that way). Some people version under Git their entire user dir.

> Tell me how my lack of ECC is going to really make life miserable here?

Git will break if RAM is bad just like anything else. That it checksums everything won't save you from checking in corrupt data, the filesystem itself being corrupt, or some internal git structure becoming corrupt. Losing your repo because something in it was written wrong is very much a possibility.

Having multiple machines involved helps, but it's not a complete fix, because the possibility exists of something damaged being transmitted from a broken machine to a good one, ensuring there's no good copy anywhere.

There's really nothing software can do to operate correctly with bad RAM all of the time. Instructions for the software are in RAM. The OS that the software expects to behave right is in RAM. Various buffers used for disk access and networking are in RAM. An application like git assumes all of that is performing correctly, and can't compensate for every possible malfunction that could happen.

Re: Non-ECC memory corrupted my hard drive image [video]

#13

Post by moloch-hai »

ubercow13 wrote: post #11
tasubotadas wrote: post #7

Earlier quoted context omitted.

When people complain about these random OS crashes and freezes it's usually RAM corruption at fault. Per 1gb of RAM, you can expect to see 266 bit errors per month[1-2] if you are using your PC 16h per day. Multiply that by 64GB or 128GB of RAM and it's crazy to think that you won't run into any of the stability issues. [1-2] https://static.googleusercontent.com/media/research.google.c... [1-2] https://en.wikipedia.o…

>266 bit errors per month That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?

Because memtest only looks at values it wrote out very recently, before they have had a chance to flip.

Memtest is looking for reliable failures, not evanescent one-off events.

Re: Non-ECC memory corrupted my hard drive image [video]

#14

Post by H8crilA »

This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You can remedy it a bit (like with ECC), but ultimately there will always be corruption if you process hundreds of petabytes of data. Get used to it, your computer executes an instruction with a probability extremely close to 1, but not equal to 1.

Deep in the archives of a well known tech company is a very well documented case of a bit flip that caused the wrong function to be executed in a C++ v-table. The big oof was that this function was the equivalent of an SQL "drop table", and just happened to be 32 bytes off of a very benign function that did something like stat(). Really funny stuff once the crisis is over :)

Re: Non-ECC memory corrupted my hard drive image [video]

#15

Post by kevingadd »

TacticalCoder wrote: post #2

> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's simply not that useful. Seriously: which percentage of developers have ECC on their development machine(s)? As developers we live in a world of SSH, cryptographic hashes, checksums everywhere, Git repositories (that is a big one), Merkle trees, digital signatures…

I went out of my way for ECC because losing files can mean losing days of work. With how much a sweng gets paid it's worth the ECC premium if it saves me a few days of work over the life of the machine.

Recently I discovered that one of my SSDs was quietly failing without setting off any warnings; doing a chkdsk showed that some files had already gotten corrupted. One of them was my backblaze backup index!

Even though I have automated backups (backblaze + macrium backups to a NAS), recovering files from them is non-trivial. If I were to lose work to non-ECC ram who knows how long it would take me to reconstruct a known-good work environment and file set. Imagine if you're working on something huge and hard to validate like neural net weights where corruption can occur silently and be hard to detect after it's happened?

Re: Non-ECC memory corrupted my hard drive image [video]

#16

Post by water8 »

H8crilA wrote: post #14

This problem has no ultimate solution. I've seen all components flip bits, CPUs, networking cards, RAM, most often you just can't know for sure what did it. You can remedy it a bit (like with ECC), but ultimately there will always be corruption if you process hundreds of petabytes of data. Get used to it, your computer executes an instruction with a probability extremely close to 1, but not equal to 1. Deep in the ar…

[dead]

Re: Non-ECC memory corrupted my hard drive image [video]

#17

Post by [deleted] »

moloch-hai wrote: post #13
ubercow13 wrote: post #11

Earlier quoted context omitted.

>266 bit errors per month That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?

Because memtest only looks at values it wrote out very recently, before they have had a chance to flip. Memtest is looking for reliable failures, not evanescent one-off events.

[deleted]

Re: Non-ECC memory corrupted my hard drive image [video]

#19

Post by eternityforest »

moloch-hai wrote: post #9

We don't have ECC mainly because Intel has long been hostile to "consumer" access to ECC. Apparently this was conceived as a market segmentation scheme: people outfitting servers could get ECC when they pay a huge premium. They would thereby not be tempted to cheap out and buy consumer-grade equipment, otherwise wholly adequate to meet all their needs at a radically cheaper price. That we cannot get laptops or even d…

I used to be a big fan of Intel, up until the latest chips from other companies that seem to have beat them on performance/watt. My next laptop will probably be AMD if the situation hasn't changed.

Re: Non-ECC memory corrupted my hard drive image [video]

#20

Post by willis936 »

moloch-hai wrote: post #13
ubercow13 wrote: post #11

Earlier quoted context omitted.

>266 bit errors per month That seems like an overestimate. How can memtest ever pass on non-ECC RAM if errors are that frequent?

Because memtest only looks at values it wrote out very recently, before they have had a chance to flip. Memtest is looking for reliable failures, not evanescent one-off events.

SDRAM is continuously refreshing all cells. How long ago data was written doesn't make a big difference (aside from the case where you're reading data immediately after writing or reading that data).
Post reply on HN