Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

blog.robertelder.org

Post reply on HN
Print view View original

51–60 of 79 posts

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#51

Post by iopq »

toast0 wrote: post #48
iopq wrote: post #45

Earlier quoted context omitted.

I've had random crashes before that were one-off. Like my desktop just froze, and then it never happened again. It only makes sense if it was a random bit-flip. The actual RAM speed never mattered, you can't tell the difference with 150 FPS and 165 FPS (even though my screen's refresh rate is 280Hz)

It's pretty easy for those to be race conditions, too. Plenty of one time crashes in a fleet of thousands of machines with ECC. ECC lets you know it's almost certainly not a memory issue. 10% more fps doesn't matter at 150 fps, but it's nice when your FPS is lower. 60 -> 66 might mean you don't dip below 60 as often. 55 -> 60.5 is pretty nice too. Maybe less of a deal if you've got VRR etc.

What game runs at only 60 FPS because of a RAM issue? I know I only have a 3600, but if a game is running at 60 FPS it's 99% because of my GPU, not the RAM.

Most games still run at 90+ FPS, I would love to have ECC RAM to prevent a potential one-off crash or just to know that the RAM didn't report an error when it happened. I would pay money for this!

Better yet, the 3d cache CPUs don't care about RAM speed as much, according to benchmarks

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#52

Post by AnthonyMouse »

jchw wrote: post #50
AnthonyMouse wrote: post #28

Earlier quoted context omitted.

ECC is error correcting. A bit gets flipped and it not only detects it but fixes it. Two bits get flipped and it can at least detect it and panic the machine immediately instead of corrupting your data. Without it the corruption is silent. Then this kind of thing happens: https://news.ycombinator.com/item?id=35026440 Which is another reason not to solder the storage either. Suppose you have a system board with bad so…

>ECC is error correcting. A bit gets flipped and it not only detects it but fixes it. Two bits get flipped and it can at least detect it and panic the machine immediately instead of corrupting your data. I did neglect to mention that ECC by-definition can correct errors, but I wonder if what's making people upset with my comment is the implication that ECC can't detect all errors. But it's true: ECC can't detect all…

Maybe the issue is that undetectable errors are possible, but if the system is in such a bad way that they're happening at any rate, you'll also be getting quite a lot of the detectable ones and then get prompt notice that something is wrong.

Whereas without ECC you could have silent data corruption for years and only discover it after it gets severe enough to warrant a manual investigation, after the damage has already propagated to your backups.

> Of course, the majority of desktops use non-ECC RAM and it's mostly fine, so I assume this is only ever going to matter in production workloads, and exactly what impact it has is hard to gauge.

There are two reasons it's useful. One is the cosmic ray random bit flip that happens even to hardware in good condition, and then ECC can usually detect and correct it, but that's less common and more important for production workloads.

The other is, your hardware is experiencing a higher than average number of random bit flips, and then ECC gives you immediate notice when this starts happening instead of letting it sow chaos until something crashes so hard you take notice.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#53

Post by NavinF »

iopq wrote: post #45
xwdv wrote: post #16

Earlier quoted context omitted.

I think you overstate the problem here. Chances are, unless you’ve addressed other more pertinent issues, simply using ECC memory isn’t going to stop systems from crashing in the middle of the night.

I've had random crashes before that were one-off. Like my desktop just froze, and then it never happened again. It only makes sense if it was a random bit-flip. The actual RAM speed never mattered, you can't tell the difference with 150 FPS and 165 FPS (even though my screen's refresh rate is 280Hz)

> you can't tell the difference with 150 FPS and 165 FPS

First byte latency makes cache misses significantly slower which in turn makes 99%ile latency (which is perceived as microstutter by humans) significantly higher even if it doesn't affect throughput (fps). This was well documented way back when the first DDR5 sticks came out and they performed like crap compared to overclocked B-Die DDR4.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#54

Post by gduihfjvffh »

johnklos wrote: post #4

ECC is good, and I genuinely wish it were more common. Thankfully, Ryzen CPUs support ECC by default (except for pre-7000 series with integrated graphics that aren't "Pro" versions), so long as the motherboard does, too (like all ASRock that I've seen). I'm running several Ryzen servers with ECC. On the other hand, there are many, many systems out there that don't have ECC, nor do they have the option to have ECC. Wh…

not just system support. availability of modules is bad.

got a HP that have both an AMD pro apu and ddr5 slots, with no soldered ram. i e. all the requirements.

it was $500 to 1500 depending on configuration. then 16 or 32gb of ecc sodimm runs over $2000 for regular consumers! and that's if you can find them in stock!

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#55

Post by chinaman425 »

silisili wrote: post #13
BearOso wrote: post #10

Earlier quoted context omitted.

I remember reading somewhere about that talk being debunked. Maybe someone more resourceful than me can find it. It was something about being more likely to be a human typo or a config change that rolled out to a bunch of machines. The statistics didn't add up, and it wasn't plausible that bit flips caused it.

Unsure about the talk, but I read a research report from someone at I think it was Cisco who purchased a second level tld of a bitsquatted US state( think statenXX.us instead of state.XX.us ). The amount of email they received was staggering, and I'd have trouble believing that many people would make that mistake in typing.

[dead]

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#56

Post by norenh »

sedivy94 wrote: post #36
johnklos wrote: post #4

ECC is good, and I genuinely wish it were more common. Thankfully, Ryzen CPUs support ECC by default (except for pre-7000 series with integrated graphics that aren't "Pro" versions), so long as the motherboard does, too (like all ASRock that I've seen). I'm running several Ryzen servers with ECC. On the other hand, there are many, many systems out there that don't have ECC, nor do they have the option to have ECC. Wh…

ECC is now baked into the DDR5 spec. Great news!

While technically true, that DDR5 comes with "on-die ECC", it is only because the memory is so unreliable it will not work properly without it. However, even then, it is not the same as true ECC that have a extra data correction chip on the memory module and also protects against send errors to the CPU.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#57

Post by p1necone »

johnklos wrote: post #4

ECC is good, and I genuinely wish it were more common. Thankfully, Ryzen CPUs support ECC by default (except for pre-7000 series with integrated graphics that aren't "Pro" versions), so long as the motherboard does, too (like all ASRock that I've seen). I'm running several Ryzen servers with ECC. On the other hand, there are many, many systems out there that don't have ECC, nor do they have the option to have ECC. Wh…

10% performance difference in exchange for maybe crashing slightly more often would be huge for people who only really use their PCs for gaming.

HN readers seem to have a skewed idea of how useful ECC is while pretending the downsides don't exist. Not everyone is primarily using their system as a workstation.

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#58

Post by undersuit »

moremetadata wrote: post #6

Moral of the story? Upgrade to DDR5 ram the latest standard which has on-die ECC memory but is not as good at spotting bit flips unlike proper ECC memory with a separate extra data correction chip. https://en.wikipedia.org/wiki/DDR5_SDRAM#:~:text=Unlike%20DD... . Whilst Proper ECC ram chips and motherboards exist, I'm surprised that a cheaper but equally as good as Proper ECC solution doesn't exist although I know so…

Intel and Asrock released a NUC with in-band ECC, equally as good at protecting your data but with performance hits.

https://www.anandtech.com/show/18732/asrock-industrial-nucs-...

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#59

Post by rozab »

hayst4ck wrote: post #2

Bit flips are totally real, at scale you will definitely see them on large queries. There was a fun talk at DEFCON on bitsquatting, the process of buying 1 bit off domain names and then accepting all incoming connections. Attacks like rowhammer similarly abuse erroneous bit flips. Supposedly microsoft can detect solar activity based on the number of windows crash logs they receive. DEFCON Talk: https://www.youtube.co…

There was an interesting post from Mozilla about this. There's a certain class of telemetry errors which seem to only be caused by cosmic rays

https://blog.mozilla.org/data/2022/04/13/this-week-in-glean-...

Re: Imaging a Hard Drive with Non-ECC Memory – What Could Go Wrong?

#60

Post by mnw21cam »

Two things. Firstly, I don't think any conclusions can be made about whether dd or dd-rescue is more susceptible to bit flips. It could be that both allocated a buffer, and dd-rescue just happened to be handed the area of memory with the fault in it, which it reused multiple times, where when dd was run that area of memory was used by something else. Memory mapping and usage in a real operating system is highly non-deterministic due to the sheer amount of things that affect it.

Secondly, once a good list of known faulty memory addresses had been created by memtest, one can tell the operating system not to use them. Then you can keep using your old hardware without the reliability problems. Although, it is possible that further areas of memory will subsequently fail, and without ECC, you'll still be vulnerable to random (cosmic ray-induced) bit flips.

Post reply on HN