Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

louwrentius.com

Post reply on HN
Print view View original

61–70 of 314 posts

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#61

Post by 8n4vidtmkvmk »

I run a similar but less sophisticated setup. About 18 TiB now, and I run it 16 hours a day. I let it sleep 8 hours per night so that it's well rested in the morning. I just do this on a cron because I'm not clever enough to SSH into a turned off (and unplugged!) machine.

4 drives: 42k hours (4.7 years), 27k hours (3 years), 15k hours (1.6 years), and the last drive I don't know because apparently it isn't SMART.

0 errors according to scrub process.

... but I guess I can't claim 0 HDD failures. There has been 1 or 2, but not for years now. Knock on wood. No data loss because of mirroring. I just can't lose 2 in a pair. (Never run RAID5 BTW, lost my whole rack doing that)

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#62

Post by lvl155 »

I have a similar approach but I don’t use ZFS. It’s a bit superfluous especially if you’re using your storage periodically (turn on and off). I use redundant NVMEs in two stages and periodically save important data into multiple HDDs (cold storage). Worth noting, it’s important to prune your data.

I also do not backup photos and videos locally. It’s a major headache and they just take up a crap ton of space when Amazon Prime will give you photo storage for free.

Anecdotally, only drives that failed on me were enterprise-grade HDDs. And they all failed within a year and in an always-on system. I also think RAIDs are over-utilized and frankly a big money pit outside of enterprise-level environments.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#63

Post by CTDOCodebases »

turnsout wrote: post #8

I’ve heard the exact opposite advice (keep the drives running to reduce wear from power cycling). Not sure what to believe, but I like having my ZFS NAS running so it can regularly run scrubs and check the data. FWIW, I’ve run my 4 drive system for 10 years with 2 drive failures in that time, but they were not enterprise grade drives (WD Green).

I think a lot of the advice around keeping the drives running is about avoiding wear caused by spin downs and startups i.e. keeping the "Start Stop Cycles" low.

Theres a difference between spinning a drive up/down once or twice a day and spinning it down every 15 minutes or less.

Also WD Green drives are not recommended for NAS usage. I know in the past they used to park the read/write head every few seconds or so which is fine if data is being accessed infrequently but continuously however a server this can result in continuous wear which leads to premature failure.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#66

Post by rincebrain »

louwrentius wrote: post #46
Jedd wrote: post #42

Surprised to not find 'ecc' on that page. I know it's not a guarantee of no-corruption, and ZFS without ECC is probably no more dangerous than any other file system without ECC, but if data corruption is a major concern for you, and you're building out a pretty hefty system like this, I can't imagine not using ECC. Slow on-disk data corruption resulting from gradual and near-silent RAM failures may be like doing regu…

The system is using ECC and I specifically - unrelated to ZFS - wanted to use ECC memory to reduce risk of data/fs corruption. I've also added 'ecc' to the original blog post to clarify. Edit: ZFS for home usage doesn't need a ton of RAM as far as I've learned. There is the 1 GB of RAM per 1TB of storage rule of thumb, but that was for a specific context. Maybe the ill-fated data deduplication feature, or was it just…

It was a handwavey rule of estimation for dedup, handwavey because dedup scales on number of records, which is going to vary wildly by recordsize.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#67

Post by CTDOCodebases »

louwrentius wrote: post #16
mvanbaak wrote: post #9

the 'secret' is not that you turn them off. it's simply luck. I have 4TB HGST drives running 24/7 for over a decade. ok, not 24 but 8, and also 0 failures. But I'm also lucky, like you. Some of the people I know have several RMAs with the same drives so there's that. My main question is: What is it that takes 71TB but can be turned off most of the time? Is this the server you store backups?

It can be luck, but with 24 drives, it feels very lucky. Somebody with proper statistics knowledge can probably calculate the risk with a guestimated 1% yearly failure rate how likely it would be to have all 24 drives remaining. And remember, my previous NAS with 20 drives also didn't have any failures. So N=44, how lucky must I be? It's for residential usage, and if I need some data, I often just copy it over 10Gbit…

I'm curious what the "Stop/Stop cycle count" is on these drives and roughly how many times per week/day you are accessing the server.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#68

Post by BenjiWiebe »

8n4vidtmkvmk wrote: post #61

I run a similar but less sophisticated setup. About 18 TiB now, and I run it 16 hours a day. I let it sleep 8 hours per night so that it's well rested in the morning. I just do this on a cron because I'm not clever enough to SSH into a turned off (and unplugged!) machine. 4 drives: 42k hours (4.7 years), 27k hours (3 years), 15k hours (1.6 years), and the last drive I don't know because apparently it isn't SMART. 0 e…

Looks like you're quite clever actually, if you can get cron to run on a powered off unplugged machine.

I think I'm missing something.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#69

Post by foobarian »

turnsout wrote: post #8

I’ve heard the exact opposite advice (keep the drives running to reduce wear from power cycling). Not sure what to believe, but I like having my ZFS NAS running so it can regularly run scrubs and check the data. FWIW, I’ve run my 4 drive system for 10 years with 2 drive failures in that time, but they were not enterprise grade drives (WD Green).

> regularly run scrubs and check the data

Does this include some kind of built-in hash/checksum system to record e.g. md5 sums of each file and periodically test them? I have a couple of big drives for family media I'd love to protect with a bit more assurance than "the drive did not fail".

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#70

Post by Filligree »

foobarian wrote: post #69
turnsout wrote: post #8

I’ve heard the exact opposite advice (keep the drives running to reduce wear from power cycling). Not sure what to believe, but I like having my ZFS NAS running so it can regularly run scrubs and check the data. FWIW, I’ve run my 4 drive system for 10 years with 2 drive failures in that time, but they were not enterprise grade drives (WD Green).

> regularly run scrubs and check the data Does this include some kind of built-in hash/checksum system to record e.g. md5 sums of each file and periodically test them? I have a couple of big drives for family media I'd love to protect with a bit more assurance than "the drive did not fail".

It's ZFS, so that's built-in. A scrub does precisely that.
Post reply on HN