Live data from Hacker News

Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

blog.robertelder.org

41–50 of 79 posts

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#41

Earlier quoted context omitted.

Totally random.

Could easily be software or some other marginal hardware bug though.

Indeed. Although, sometimes the machine wouldn't fully crash. It was like the disk was corrupted, but apps were still running, which makes me suspect it was the lack of ECC.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#42
post #4

ECC is good, and I genuinely wish it were more common. Thankfully, Ryzen CPUs support ECC by default (except for pre-7000 series with integrated graphics that aren't "Pro" versions), so long as the motherboard does, too (like all ASRock that I've seen). I'm running several Ryzen servers with ECC. On the other hand, there are many, many systems out there that don't have ECC, nor do they have the option to have ECC. Wh…

ECC is now baked into the DDR5 spec. Great news!

On-Chip ECC, it's an improvement but full ECC memory, which you can get for DDR5, also protects your data in transmission at 6400MT/s.

Additionally the on-chip EEC of DDR5 won't report the errors to your OS. ECC memory errors when corrected can be handled by the OS, and you'll even be informed of the uncorrectable 2 bit errors.

Want to protect for 2-bit errors? Make sure your platform has support for ECC-chipkill.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#43
post #2

Bit flips are totally real, at scale you will definitely see them on large queries. There was a fun talk at DEFCON on bitsquatting, the process of buying 1 bit off domain names and then accepting all incoming connections. Attacks like rowhammer similarly abuse erroneous bit flips. Supposedly microsoft can detect solar activity based on the number of windows crash logs they receive. DEFCON Talk: https://www.youtube.co…

One of my favorite DEFCON talks of all time.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#44
post #7

I've had a lot of really strange bugs and data loss with my current build (Ryzen with Gskill memory). After running a memtest for 24h i finally saw that two of the four ram sticks were faulty (two bit flips on each only rarely and on a specific test). The company changed them but now a year later without any issues I have another one that failed in exactly the same way. This is the last time I build a non-ECC system…

My motherboard isn't rated for more than DDR4-3200 with my old cpu, a Ryzen 7 2700. I could set my memory's XMP profile up and run at DDR4-3466 and memtest would be stable for more than 24 hours but would error before 48. I backed off, DDR-3400, DDR4-3333, DDR4-3266... finally stable in memtest for 96 hours, boot into Windows and run Prime95 Blend workload, 3266 crashes in hours. I finally find a little note in my motherboard manual that older CPUs are limited to DDR4-3200. Set that speed, rock solid, I was even able to tighten the JEDEC timings with guidance from the second XMP profile for DDR4-3133.

Gigabyte really did mean DDR4-3200 was the limit for Pinnacle Ridge and older AMD cpus.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#45
post #16
post #4

ECC is good, and I genuinely wish it were more common. Thankfully, Ryzen CPUs support ECC by default (except for pre-7000 series with integrated graphics that aren't "Pro" versions), so long as the motherboard does, too (like all ASRock that I've seen). I'm running several Ryzen servers with ECC. On the other hand, there are many, many systems out there that don't have ECC, nor do they have the option to have ECC. Wh…

I think you overstate the problem here. Chances are, unless you’ve addressed other more pertinent issues, simply using ECC memory isn’t going to stop systems from crashing in the middle of the night.

I've had random crashes before that were one-off.

Like my desktop just froze, and then it never happened again. It only makes sense if it was a random bit-flip.

The actual RAM speed never mattered, you can't tell the difference with 150 FPS and 165 FPS (even though my screen's refresh rate is 280Hz)

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#46
post #4

ECC is good, and I genuinely wish it were more common. Thankfully, Ryzen CPUs support ECC by default (except for pre-7000 series with integrated graphics that aren't "Pro" versions), so long as the motherboard does, too (like all ASRock that I've seen). I'm running several Ryzen servers with ECC. On the other hand, there are many, many systems out there that don't have ECC, nor do they have the option to have ECC. Wh…

Intel finally started supporting ECC again on their consumer CPUs, but if only the W680 didn't cost ~$400 starting.

[deleted]

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#47
post #29
post #5

I ran a cluster of ~30k blade based computers booting entirely off iPXE. They didn't have any onboard ssd/disk storage or ECC memory. Every day, a few of them would randomly lock up, they'd reboot with a fresh network image and keep on humming.

How do you even get that many computers without ECC? I think all the blades I've seen have ECC as baseline spec.

Google started with "standard PCs", though I'm not finding any info if they used ECC memory.

https://blog.codinghorror.com/building-a-computer-the-google...

https://patents.google.com/patent/US6549988

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#48
post #45
post #16

Earlier quoted context omitted.

I think you overstate the problem here. Chances are, unless you’ve addressed other more pertinent issues, simply using ECC memory isn’t going to stop systems from crashing in the middle of the night.

I've had random crashes before that were one-off. Like my desktop just froze, and then it never happened again. It only makes sense if it was a random bit-flip. The actual RAM speed never mattered, you can't tell the difference with 150 FPS and 165 FPS (even though my screen's refresh rate is 280Hz)

It's pretty easy for those to be race conditions, too. Plenty of one time crashes in a fleet of thousands of machines with ECC. ECC lets you know it's almost certainly not a memory issue.

10% more fps doesn't matter at 150 fps, but it's nice when your FPS is lower. 60 -> 66 might mean you don't dip below 60 as often. 55 -> 60.5 is pretty nice too. Maybe less of a deal if you've got VRR etc.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#49
post #4

ECC is good, and I genuinely wish it were more common. Thankfully, Ryzen CPUs support ECC by default (except for pre-7000 series with integrated graphics that aren't "Pro" versions), so long as the motherboard does, too (like all ASRock that I've seen). I'm running several Ryzen servers with ECC. On the other hand, there are many, many systems out there that don't have ECC, nor do they have the option to have ECC. Wh…

Crashes in the middle of the night are not what worries me. Who cares. It's silent data loss that can go unnoticed for a very long time. And not just a single bit. If the flip hits file system structures or file layout you can have massive silent data loss.

Yep. It's why ZFS, BTRFS, Ceph and Gluster matter. Being able to detect that data at rest has gone wrong, and being able to reconstruct the original state is a big deal.

I'd like to think that as NAND continues to scale up in capacity and lower in cost, that we'll see some real shakeup to filesystems and storage where self-healing mass storage can be genuinely commoditized -- not something that's only accessible to businesses (and computing enthusiasts) due to cost and complexity.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#50
post #18

Earlier quoted context omitted.

Well, the reason why ECC mattered here is because the RAM was bad, but modern Mac computers do not come with user-serviceable RAM at all, so if you have a problem like this, it's a support ticket anyways, and I'm not even sure there's a true equivalent to Memtest86 for modern Mac computers in the first place. So basically, if it was a RAM problem, there's no point in diagnosing it even if you could; just send your Ma…

ECC is error correcting. A bit gets flipped and it not only detects it but fixes it. Two bits get flipped and it can at least detect it and panic the machine immediately instead of corrupting your data. Without it the corruption is silent. Then this kind of thing happens: https://news.ycombinator.com/item?id=35026440 Which is another reason not to solder the storage either. Suppose you have a system board with bad so…

>ECC is error correcting. A bit gets flipped and it not only detects it but fixes it. Two bits get flipped and it can at least detect it and panic the machine immediately instead of corrupting your data.

I did neglect to mention that ECC by-definition can correct errors, but I wonder if what's making people upset with my comment is the implication that ECC can't detect all errors.

But it's true: ECC can't detect all bitflips, and in fact there's at least one study[1] that suggests quite a lot of memory errors go entirely undetected even with ECC.

Silent corruption does in fact occur even with ECC and it may not even be particularly rare, even though it is rarer than typical single/double-bit flips. Of course, the majority of desktops use non-ECC RAM and it's mostly fine, so I assume this is only ever going to matter in production workloads, and exactly what impact it has is hard to gauge.

[1]: https://pages.cs.wisc.edu/~remzi/Classes/739/Fall2018/Papers...

Post reply on HN