Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

blog.robertelder.org

Post reply on HN
Print view View original

21–30 of 79 posts

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#21

Post by ilyt »

That reminds me of how I accidentally tracked memory issue to the failing power supply.

I noticed (after some windows bluescreen) on memtest that the memory is showing some errors. Ordered another 16GB pair, replaced it and.... the problem persisted.

Suspecting something with motherboard I just chalked it to something with mobo and pretty much said "well I'm not replacing mobo now, it will have to wait for next hardware refresh. Gaming PC so no big deal. And now I had 32 GB of RAM in PC.

Weirdly enough, problem only happened when running on multi-core memory test.

Cue ~1 year after and my power supply just... died. Guessing bad caps I just ordered another and thought nothing of it. On a whim I ran memtest and....

nothing. All fixed. Repeated few times and it was just fine, no bluescreen for ~ 2 years now too.

I definitely want to get next machine with ECC but the DDR4 consumer ECC situation looks... weird. I'm not sure whether I should be happy with on-chip ECC, I'd really prefer to have whole CPU-memory pipe ECCed

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#22

Post by moremetadata »

BearOso wrote: post #14
moremetadata wrote: post #6

Moral of the story? Upgrade to DDR5 ram the latest standard which has on-die ECC memory but is not as good at spotting bit flips unlike proper ECC memory with a separate extra data correction chip. https://en.wikipedia.org/wiki/DDR5_SDRAM#:~:text=Unlike%20DD... . Whilst Proper ECC ram chips and motherboards exist, I'm surprised that a cheaper but equally as good as Proper ECC solution doesn't exist although I know so…

DDR5 has enough ECC on chip to make errors effectively impossible. It doesn't provide error data to the CPU, though, so errors in transit can still occur. This is really unlikely, though, and anything not mission-critical will no longer need the extra ECC computation on the CPU-side. (DDR5 encapsulates the memory controller).

> This is really unlikely, though,

I think you can say that because people are not routinely monitoring their surroundings for ionizing radiation.

If this were to change, I think we can start to identify some of those military locations which could be interfering with equipment, that would then expose the weakness of DDR5.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#23

Post by ilyt »

BearOso wrote: post #14
moremetadata wrote: post #6

Moral of the story? Upgrade to DDR5 ram the latest standard which has on-die ECC memory but is not as good at spotting bit flips unlike proper ECC memory with a separate extra data correction chip. https://en.wikipedia.org/wiki/DDR5_SDRAM#:~:text=Unlike%20DD... . Whilst Proper ECC ram chips and motherboards exist, I'm surprised that a cheaper but equally as good as Proper ECC solution doesn't exist although I know so…

DDR5 has enough ECC on chip to make errors effectively impossible. It doesn't provide error data to the CPU, though, so errors in transit can still occur. This is really unlikely, though, and anything not mission-critical will no longer need the extra ECC computation on the CPU-side. (DDR5 encapsulates the memory controller).

source on that ? Did anyone tested that ?

> This is really unlikely, though, and anything not mission-critical will no longer need the extra ECC computation on the CPU-side.

ECC computation is done in hardware anyway

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#25

Post by teddyh »

moremetadata wrote: post #6

Moral of the story? Upgrade to DDR5 ram the latest standard which has on-die ECC memory but is not as good at spotting bit flips unlike proper ECC memory with a separate extra data correction chip. https://en.wikipedia.org/wiki/DDR5_SDRAM#:~:text=Unlike%20DD... . Whilst Proper ECC ram chips and motherboards exist, I'm surprised that a cheaper but equally as good as Proper ECC solution doesn't exist although I know so…

Also from your link:

“There still exist non-ECC and ECC DDR5 DIMM variants; the ECC variants have extra data lines to the CPU to send error-detection data, letting the CPU detect and correct errors that occurred in transit.”

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#26

Post by thejosh »

johnklos wrote: post #4

ECC is good, and I genuinely wish it were more common. Thankfully, Ryzen CPUs support ECC by default (except for pre-7000 series with integrated graphics that aren't "Pro" versions), so long as the motherboard does, too (like all ASRock that I've seen). I'm running several Ryzen servers with ECC. On the other hand, there are many, many systems out there that don't have ECC, nor do they have the option to have ECC. Wh…

Depending on your scale, software running on the machine is more likely to cause crashes in the middle of the night :)

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#27

Post by russdill »

johnklos wrote: post #4

ECC is good, and I genuinely wish it were more common. Thankfully, Ryzen CPUs support ECC by default (except for pre-7000 series with integrated graphics that aren't "Pro" versions), so long as the motherboard does, too (like all ASRock that I've seen). I'm running several Ryzen servers with ECC. On the other hand, there are many, many systems out there that don't have ECC, nor do they have the option to have ECC. Wh…

Crashes in the middle of the night are not what worries me. Who cares. It's silent data loss that can go unnoticed for a very long time. And not just a single bit. If the flip hits file system structures or file layout you can have massive silent data loss.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#28

Post by AnthonyMouse »

jchw wrote: post #18
muro wrote: post #15

AFAICT, no current Mac comes with ECC - do they have the same issues? If so, one doesn't hear about them too often.

Well, the reason why ECC mattered here is because the RAM was bad, but modern Mac computers do not come with user-serviceable RAM at all, so if you have a problem like this, it's a support ticket anyways, and I'm not even sure there's a true equivalent to Memtest86 for modern Mac computers in the first place. So basically, if it was a RAM problem, there's no point in diagnosing it even if you could; just send your Ma…

ECC is error correcting. A bit gets flipped and it not only detects it but fixes it. Two bits get flipped and it can at least detect it and panic the machine immediately instead of corrupting your data.

Without it the corruption is silent. Then this kind of thing happens:

https://news.ycombinator.com/item?id=35026440

Which is another reason not to solder the storage either.

Suppose you have a system board with bad soldered memory and you want to copy your data off of it onto the new one. Well, the memory is flipping random bits as it's copying, but the flash chips are permanently attached to the same board as the bad memory.

Otherwise it would have been just a support ticket; now it's something worse.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#29

Post by jwiz »

latchkey wrote: post #5

I ran a cluster of ~30k blade based computers booting entirely off iPXE. They didn't have any onboard ssd/disk storage or ECC memory. Every day, a few of them would randomly lock up, they'd reboot with a fresh network image and keep on humming.

How do you even get that many computers without ECC? I think all the blades I've seen have ECC as baseline spec.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#30

Post by AnthonyMouse »

BearOso wrote: post #14
moremetadata wrote: post #6

Moral of the story? Upgrade to DDR5 ram the latest standard which has on-die ECC memory but is not as good at spotting bit flips unlike proper ECC memory with a separate extra data correction chip. https://en.wikipedia.org/wiki/DDR5_SDRAM#:~:text=Unlike%20DD... . Whilst Proper ECC ram chips and motherboards exist, I'm surprised that a cheaper but equally as good as Proper ECC solution doesn't exist although I know so…

DDR5 has enough ECC on chip to make errors effectively impossible. It doesn't provide error data to the CPU, though, so errors in transit can still occur. This is really unlikely, though, and anything not mission-critical will no longer need the extra ECC computation on the CPU-side. (DDR5 encapsulates the memory controller).

> This is really unlikely, though

It happens quite often as a result of dust in the contacts when the memory was installed or weak solder on the chips or sockets or bad capacitors etc.

None of which is that likely on machines in good working order, but many are not. And you can go from one to the other at any time as a result of a power spike or a cooling failure.

Post reply on HN