Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Non-ECC memory corrupted my hard drive image [video]

youtube.com

Post reply on HN
Print view View original

151–160 of 161 posts

Re: Non-ECC memory corrupted my hard drive image [video]

#151

Post by dvaletin »

Filligree wrote: post #95
Gordonjcp wrote: post #93

Earlier quoted context omitted.

> That we cannot get laptops or even desk machines with ECC, and so have them crash frequently, is seen as a trivial side effect of the strategy. How frequently would you say you encounter a crash that you can pin down to a lack of ECC memory in your laptop or desktop?

You can't, that's the thing, right? I have a Ryzen desktop with ECC, and it registers about one bit-flip per week. I don't know how many of those would become crashes, but I'm more worried about the ones that wouldn't.

interesting to see the coincidence with solar flares

Re: Non-ECC memory corrupted my hard drive image [video]

#152

Post by nirvdrum »

Too wrote: post #141
nirvdrum wrote: post #118

Earlier quoted context omitted.

How do you confirm that there's no corruption? My understanding was a silent bit flip would be difficult to detect. I ended up using ZFS and ECC memory to avoid silent corruption, but I'm largely relying on the motherboard to do the right thing when an ECC error is detected. It'd be nice to verify after the fact.

git fsck

On 932 GB of binary data? I'm probably operating on old impressions of git, but that sounds like it'd be a nightmare to work with. Ideally, I'd like something that runs in a reasonable amount of time so it can be run on a regular schedule. I currently run a ZFS scrub that takes about 90 hours to finish on 32 TB of data, so it's only run monthly.

Re: Non-ECC memory corrupted my hard drive image [video]

#153

Post by justinclift »

to11mtm wrote: post #149
justinclift wrote: post #143

Near the end, the video mentions his (old) computer "apparently can't take 'high density' memory". Then the author goes through a process of randomly buying memory and hoping it works. Anyone know what that "high density" memory problem is about? Maybe it's a misunderstanding of memory channels and ranks?

Not sure if it's what he means, but in my experience 'High density' refers to less chips for a given capacity. When I used to sling PC hardware, as memory densities increased, you would run into things like 'This Motherboard will only take a 256MB module if it has 8 chips on both sides (16 chips total), it will not take a 256MB module with 8 chips on one side (8 chips total)' Depending on the board, it might only reg…

Interesting. Have heard mention of similar over the years, but never seen it in person myself.

Sounds strange, as - without knowing any better - I'd expect the memory interface to be agnostic to the chips implementing it. eg "address 0x11111111" on 'high density' ram would be seen exactly the same as "address 0x11111111" on other ram

The mind boggles. :)

Re: Non-ECC memory corrupted my hard drive image [video]

#154

Post by moloch-hai »

layer8 wrote: post #86
gruez wrote: post #55

Earlier quoted context omitted.

I don't doubt that non-ECC hardware experiences some non-zero number of bitflips per year. I'm just doubting the parent commenter's claim that non-ECC ram is causing computers to crash "frequently".

And the parent is pointing out that not crashing on bit flips is exactly the problem.

Actually crashing on bit flips is a second problem, and an indicator of the first problem.

Re: Non-ECC memory corrupted my hard drive image [video]

#155

Post by simoncion »

encryptluks2 wrote: post #138
simoncion wrote: post #67

Earlier quoted context omitted.

> ...but try finding some high performance 16GB sticks of ECC DDR4 RAM like what you'll see on gaming computers. Here ya go: https://nemixram.com/16gb-ddr4-3200-pc4-25600-ecc-udimm-2rx8... It doesn't have pretty lights on it, but it does seem to be in the same speed class that gets called "gaming RAM" by a _whole_ bunch of retailers.

For real. I am going to have to look into this more. While it isn't by a global retail manufacturer, the price is like a fraction of most other RAM so I am curious. Does NewEgg or Amazon sell it directly, and if not why not?

NEMIX sells through Newegg, but it's one of those "Newegg routes the sale through the seller" things, rather than "Newegg buys the thing from the seller and handles the shipping and everything" thing. In my experience, the price is the same whether you order through Newegg's website, or through NEMIX's website.

Unless you have a particular reason to keep your order history in Newegg, just buy direct from NEMIX.

(IDK about NEMIX's relationship with Amazon, as I don't buy things from Amazon.)

Re: Non-ECC memory corrupted my hard drive image [video]

#156

Post by Brian_K_White »

moloch-hai wrote: post #154
layer8 wrote: post #86

Earlier quoted context omitted.

And the parent is pointing out that not crashing on bit flips is exactly the problem.

Actually crashing on bit flips is a second problem, and an indicator of the first problem.

Crashing is "not the problem" because crashing prevents the far worse problem of incorrect data or operation that you don't know is incorrect.

The difference in significance is so great that by comparison a mere crash is no problem at all.

In fact you design systems with hair triggers to 'crash' on purpose as readily as possible. Trying desperately to crash at all times every millisecond all day every day.

IE, halting all operation of some subsystem, or the whole thing, the second a single bit wrong is detected. Better to kill the hd or the entire machine than let it keep running one second after getting any hint it might not be 100% trustworthy.

'crash' in quotes because really all you want to do is halt, and you're doing it on purpose, but that is still a crash, in the sense that your application does not want to halt and it isn't necessarily halted gracefully with any chance to conclude anything or save anything. Those are just more operations you can no longer trust to be correct, and so should not be allowed to do.

Re: Non-ECC memory corrupted my hard drive image [video]

#157

Post by mrlonglong »

sgtnoodle wrote: post #105
mrlonglong wrote: post #90

We've got a server that keeps rebooting due to a bad ECC DIMM chip. I thought the whole point of ECC was to keep the server going until we can replace the DIMM?

It sounds like it's working as intended. Redundancy doesn't help you with reliability once your system is in a degraded state. Running a server with a known bad ram stick is like driving a car on run-flat tires. It will probably get you where you need to go, but you really should be driving to a tire shop.

Ok thanks I understand things better now. I'll pester my boss to get new DIMM asap.

Re: Non-ECC memory corrupted my hard drive image [video]

#158

Post by Nomentatus »

eek2121 wrote: post #112
gruez wrote: post #29

Earlier quoted context omitted.

> That we cannot get laptops or even desk machines with ECC, and so have them crash frequently, is seen as a trivial side effect of the strategy I’m not sure what you mean by “frequently”, but my non-ECC machines definitely do not crash “frequently”. > before MS, a program crashing was grounds for a refund Source?

I see a ton of replies here. I am a tech user with thousands of terabytes stored locally. Most of those terabytes are mirrored in the cloud. In all of those situations I've never actually had any corruption related to memory or storage. I have 932gb in PICTURES ffs. I get the claimed need for EEC, but yet I don't. I own a number of web properties and regularly update logos, software packages, html files, PHP files, r…

If your pictures are not RAW but compressed (say jpgs) then the corrupted photos will really jump out at you - half the picture is there, but then noise. I used to make sure I was storing jpgs not RAW, but then learned that wasn't always a good thing, several times over. Dunno whether damage happened in RAM or on HD, but I strongly suspect RAM.

The bad habit that nailed me was probably this one: I would ball up a whole bunch of photos in a big .zip; (to enhance copying speed and maybe privacy) then next year unzip that, add that year's selected photos and zip it again. Year after year. That added up to a lot of trips through RAM in which one bit flip could take out an image. Bugs in compression software aren't impossible as the villan, but that's not my bet.

Solution: I now zip up a few years together and copy each chunk I don't ever rezip old large chunks. No new errors noticed.

Re: Non-ECC memory corrupted my hard drive image [video]

#159

Post by Nomentatus »

nirvdrum wrote: post #118
eek2121 wrote: post #112

Earlier quoted context omitted.

I see a ton of replies here. I am a tech user with thousands of terabytes stored locally. Most of those terabytes are mirrored in the cloud. In all of those situations I've never actually had any corruption related to memory or storage. I have 932gb in PICTURES ffs. I get the claimed need for EEC, but yet I don't. I own a number of web properties and regularly update logos, software packages, html files, PHP files, r…

How do you confirm that there's no corruption? My understanding was a silent bit flip would be difficult to detect. I ended up using ZFS and ECC memory to avoid silent corruption, but I'm largely relying on the motherboard to do the right thing when an ECC error is detected. It'd be nice to verify after the fact.

bit flip hard to detect - true for RAW, false for .jpg or other compressed files.

Re: Non-ECC memory corrupted my hard drive image [video]

#160

Post by dijit »

PragmaticPulp wrote: post #103
Filligree wrote: post #95

Earlier quoted context omitted.

You can't, that's the thing, right? I have a Ryzen desktop with ECC, and it registers about one bit-flip per week. I don't know how many of those would become crashes, but I'm more worried about the ones that wouldn't.

> I have a Ryzen desktop with ECC, and it registers about one bit-flip per week. This isn’t normal. You have a bad memory module. A non-ECC machine should able to support memtest86 (a rigorous memory testing tool) for a week straight without a single bitflip. Having a bit flip per week is so far away from normal that it’s definitely a bad memory part.

> > I have a Ryzen desktop with ECC, and it registers about one bit-flip per week.

> This isn’t normal. You have a bad memory module.

> Having a bit flip per week is so far away from normal that it’s definitely a bad memory part.

It's hard to know what's normal. If you're somewhere in the world like northern Sweden the chance of cosmic rays goes up from being less protected from the suns rays (Hey, but northern lights!) and if you're at higher altitude; along with less interesting things such as how shielded the ram is (server chassis, if you removed the rack doors or not).

Memory density has a lot to go with it too, the chances of bitflips increases a lot.

Anyway; regardless of the theory behind it: I've run global operations with multiple-thousands of physical machines with high density ram (and 16-32 DIMMS in each server) and noticed at least 1 bit flip a week on average for each machine. But some locations were worse impacted than others.

Annoyingly, ECC is not created equal, there are some correctable bit-flips that are hidden from the iDRAC/iLO: this is the same type of ECC that the rPI is using (referred to as on-die): https://datasheets.raspberrypi.com/rpi4/raspberry-pi-4-produ...

As with all things like this YMMV, but ECC is doing a lot of heavy lifting.

Post reply on HN