Live data from Hacker News

Non-ECC memory corrupted my hard drive image [video]

youtube.com

51–60 of 161 posts

Re: Non-ECC memory corrupted my hard drive image [video]

#52

Earlier quoted context omitted.

Because memtest only looks at values it wrote out very recently, before they have had a chance to flip. Memtest is looking for reliable failures, not evanescent one-off events.

SDRAM is continuously refreshing all cells. How long ago data was written doesn't make a big difference (aside from the case where you're reading data immediately after writing or reading that data).

> SDRAM is continuously refreshing all cells

...so ? we're not talking about slow deterioration (which is why refresh is needed), we're talking about bit flip from cosmic rays where cell changes state completely.

Re: Non-ECC memory corrupted my hard drive image [video]

#53

> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's simply not that useful. Seriously: which percentage of developers have ECC on their development machine(s)? As developers we live in a world of SSH, cryptographic hashes, checksums everywhere, Git repositories (that is a big one), Merkle trees, digital signatures…

> Seriously: which percentage of developers have ECC on their development machine(s)? Every single one of my non-mini computers uses ECC ram except for my laptop. If someone would release a framework laptop motherboard that supports ECC ram (and preferably risc-v) I'd finally be able to close to reliability gap. It blows my mind that we say "Well sure, we COULD make infallible ram, but that would cost a tiny bit extr…

"If someone would release a framework laptop motherboard that supports ECC ram..."

This. If I could, I would. The only reason my laptop doesn't have ecc is because the manufacturer doesn't offer the option in any macines I otherwise want.

That comment was very misguided in trying to suggest that there is any valid excuse to tolerate unreliable execution hardware. git and ssh and md5sums do not mean that it's ok if your very brain can't be trusted to deliver data from one part to another within itself, or spit back the same data that was put in a cell. Everything else is built upon that!

Re: Non-ECC memory corrupted my hard drive image [video]

#54
post #29

Earlier quoted context omitted.

> That we cannot get laptops or even desk machines with ECC, and so have them crash frequently, is seen as a trivial side effect of the strategy I’m not sure what you mean by “frequently”, but my non-ECC machines definitely do not crash “frequently”. > before MS, a program crashing was grounds for a refund Source?

"frequently" is very subjective or relative in this context. 25 years ago I had a crash per hour on almost any regular computer, but zero crashes per month on servers with ECC. In the past couple of years I think I had a few cases of frozen apps, but I don't remember of any OS level problem. At the same time, on servers I see from time to time ECC fixing a bit, but on the desktop or laptop I have no idea how many tim…

>25 years ago I had a crash per hour on almost any regular computer, but zero crashes per month on servers with ECC

If it's crashing once per hour, it's probably unstable drivers/software or flaky hardware that needs to be RMAed, not random bitflips.

Re: Non-ECC memory corrupted my hard drive image [video]

#55
post #29

Earlier quoted context omitted.

> That we cannot get laptops or even desk machines with ECC, and so have them crash frequently, is seen as a trivial side effect of the strategy I’m not sure what you mean by “frequently”, but my non-ECC machines definitely do not crash “frequently”. > before MS, a program crashing was grounds for a refund Source?

The problem with untrusworthy memory (or any other component) is not that your system crashes, it's that it doesn't .

I don't doubt that non-ECC hardware experiences some non-zero number of bitflips per year. I'm just doubting the parent commenter's claim that non-ECC ram is causing computers to crash "frequently".

Re: Non-ECC memory corrupted my hard drive image [video]

#56
post #36

Earlier quoted context omitted.

I'm happy with my AMD based laptop. But I haven't seen any that support ECC. But I did see a Lenovo model, IIRC, that had some kind of Xeon and ECC. Not sure what the noise and battery life situations on that thing were, though.

I realise that's blurrier when it comes to laptops, but AIUI it's more a case of whether the motherboard than supports it than about the AMD chip. i.e. given a desktop CPU, as far as I know you can put it in a motherboard that either does or does not support ECC RAM.

Ryzen APUs, which include almost all AMD laptops, actually have ECC fused off in silicon unless you buy the "Pro" variant.

Re: Non-ECC memory corrupted my hard drive image [video]

#58
post #44
post #24

Earlier quoted context omitted.

You are under the impression that CPUs and other chips always perform the same instructions as are written in the code, and only RAM can flip bits because DRAM is DRAM :) It can (and should! whenever possible) be improved, not fixed. There's always that pesky gamma that can hit a specific transistor, even if it is deep underground. Gamma cannot be fully stopped. At certain scales data corruption becomes directly meas…

Sure but once your registers, cache, data bus and address bus has ECC you have vastly smaller area that can flip. You can even just buy (well, chipaggedon aside) ARM cores that have 2 chips running in parallel and faulting when the result is different

> You can even just buy (well, chipaggedon aside) ARM cores that have 2 chips running in parallel and faulting when the result is different

See dual-core lock-step Arm chips (used for automotive).

Re: Non-ECC memory corrupted my hard drive image [video]

#59
post #48
post #4

Earlier quoted context omitted.

But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again. The CPU, RAM and mobo manufacturers need to get together and make ECC RAM mandatory. It's absurd that we have machines with gigabytes of storage using microscopic (nanoscopic??) capacitor…

>But if I open up an editor and write a document then that could be corrupted in RAM and then the corrupted data saved to disk. The document is likely to be more important than some ripped CDs or a git repo that I can download again. Devils advocate: if it is just some bits in character flipped it's entirely recoverable while flipping some bits in compressed stream for video would corrupt more.

Agreed. One problem with RAM errors (which I've actually experienced) is they are insidious. You probably won't notice them immediately so the error can be propagated, and they're very very difficult to diagnose if you do notice them.

Back in the 90s we had a database server which had a stuck bit in memory normally mapped to the page cache. This caused sectors to be written to the backing software RAID which couldn't be read back in (because I think some checksum was corrupted when written and then failed when read back). It took an absolute age to diagnose this. I think I only worked it out by eliminating everything else.

Re: Non-ECC memory corrupted my hard drive image [video]

#60

> Why ECC Memory Is So Important Except that it's not for many use cases. It's great for servers but for people on their personal and/or work computer, it's simply not that useful. Seriously: which percentage of developers have ECC on their development machine(s)? As developers we live in a world of SSH, cryptographic hashes, checksums everywhere, Git repositories (that is a big one), Merkle trees, digital signatures…

When people complain about these random OS crashes and freezes it's usually RAM corruption at fault. Per 1gb of RAM, you can expect to see 266 bit errors per month[1-2] if you are using your PC 16h per day. Multiply that by 64GB or 128GB of RAM and it's crazy to think that you won't run into any of the stability issues. [1-2] https://static.googleusercontent.com/media/research.google.c... [1-2] https://en.wikipedia.o…

I don't think this rate is reasonable to use in this manner. I've run thousands of servers, with lots of ECC ram (quite a few servers had 768GB of ram, but most were more like 32 or 64). The vast majority ran for years with zero reported errors. A small handful would have major errors of thousands in an hour, but we would replace for hundreds in an hour. A couple servers developed a periodic report of one or two (correctable) errors per day. If 266 bit errors per month per 1Gb was a usable rate, all of our servers would have been throwing ECC correctable errors all the time.

But I didn't have time or desire to publish a study on our experience, so there's no hard numbers.

Post reply on HN