Live data from Hacker News

My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

louwrentius.com

111–120 of 314 posts

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#111
Do you have a drive rotation schedule?

24 drives. Same model. Likely the same batch. Similar wear. Imagine most of them failing at the same time, and the rest failing as you're rebuilding it due to the increased load, because they're already almost at the same point.

Reliable storage is tricky.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#112

Do you have a drive rotation schedule? 24 drives. Same model. Likely the same batch. Similar wear. Imagine most of them failing at the same time, and the rest failing as you're rebuilding it due to the increased load, because they're already almost at the same point. Reliable storage is tricky.

This. I had just two drives in raid 1, and the 2nd drive failed immediately after silvering a new drive to re-create the array. very lucky :D

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#113
post #89

Earlier quoted context omitted.

It's difficult as a home user to find ECC memory, harder to make sure it actually works in your hardware configuration, and near-impossible to find ECC memory that doesn't require lower speeds than what you can get for $50 on amazon. I would very much like to put ECC memory in my home server, but I couldn't figure it out this generation. After four hours I decided I had better things to do with my time.

Indeed. I'd started to add an aside to the effect of 'ten years ago it was probably easier to go ECC'. I'll add it here instead. A decade ago if you wanted ECC your choice was basically Xeon, and all( ) Xeon motherboards would accept ECC. I agree that these days it's much more complex, since you are ineluctably going get sucked into the despair-spiral of trying to work out what combination of Ryzen + motherboard + EC…

Sounds like the answer is to just buy another Xeon then, even if it's a little older and maybe secondhand. I think there's a reason the vast majority of Supermicro motherboards are still just Intel only.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#115

Do you have a drive rotation schedule? 24 drives. Same model. Likely the same batch. Similar wear. Imagine most of them failing at the same time, and the rest failing as you're rebuilding it due to the increased load, because they're already almost at the same point. Reliable storage is tricky.

Reminds me of the HN outage where two SSDs both failed after 40k hours: https://news.ycombinator.com/item?id=32031243

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#116
post #80

Earlier quoted context omitted.

How does dust affect things? The drives are airtight.

They're airtight now (at the high end or enterprise level). They weren't airtight not very long ago and had filters to regulate the air exchange.

They're not airtight in the true sense (besides the helium filled ones nowadays), but every drive made in the past... 30? 40 years is airtight in the sense that no dust can ever get into the drive. There's a breather hole somewhere (with a big warning to not cover it!) to equalize pressure, and a filter that doesn't allow essentially any particles in.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#117

Earlier quoted context omitted.

A drive failure every 6 months almost sounds more like dirty power than dust, I’ve always kept my NAS/file servers in dusty residential environments (I have a nice fuzzy gray Synology logo visible right now) and never seen anything like that

Drives are sealed anyway. Humidity maybe. Dust can’t really get in. Power or bad batch of drives.

Except for the helium-filled ones, they aren't sealed; there is a very fine filter that equalises atmospheric pressure. (This is also why they have a maximum operating altitude --- the head needs a certain amount of atmospheric pressure to float.)

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#118
post #8

I’ve heard the exact opposite advice (keep the drives running to reduce wear from power cycling). Not sure what to believe, but I like having my ZFS NAS running so it can regularly run scrubs and check the data. FWIW, I’ve run my 4 drive system for 10 years with 2 drive failures in that time, but they were not enterprise grade drives (WD Green).

I think a lot of the advice around keeping the drives running is about avoiding wear caused by spin downs and startups i.e. keeping the "Start Stop Cycles" low. Theres a difference between spinning a drive up/down once or twice a day and spinning it down every 15 minutes or less. Also WD Green drives are not recommended for NAS usage. I know in the past they used to park the read/write head every few seconds or so wh…

There used to be some tutorials going around to flash the firmware to turn greens into reds I believe. Which simply disables the head parking.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#119
post #25

Earlier quoted context omitted.

> There have been drives where power cycling was hazardous. I know about this story from 30+ years ago. It may have been true then. It may be even true now. Yet, in my case, I don't power cycle these drives often. At most a few times a month. I can't say or prove it's a huge risk. I only believe it's not. I have accepted this risk for over 15+ years. Update: remember that hard drives have an option to spin down when…

I debated posting because it felt like shitstirring. I think overwhelmingly what you're doing is right. And if a remote power on eg WOL works on the device, so much the better. If I could wish for one thing, it's mods to code or documentation of how to handle drive power down on zfs. The rumour mill is zfs doesn't like spindown.

What is there to handle? I have a ZFS array that works just fine with hard drives that automatically spin down. ZFS handles this without an issue.

The main gotchas tend to be: if you use the array for many things, especially stuff that throws off log files, you will constantly be accessing that array and resetting the spin down timers. Or you might be just at the threshold for spindown and you'll put a ton of cycles on it as it bounces from spindown to access to spin up.

For a static file server (rarely accessed backups or media), partitioned correctly, it works great.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#120
Regular reminder: RAID (and ZFS) don't replace backups. It's an availability solution to reduce downtime in event of disk failure. Many things can go wrong with your files and filesystem besides disk failure, eg user error, userspace software/script bugs, driver or FS or hardware bugs, ransomware, etc)

The article mentions backups near the end saying eg "most of the data is not important" and the "most important" data is backed up. Feeling lucky I guess.

Post reply on HN