Live data from Hacker News

My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

louwrentius.com

121–130 of 314 posts

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#121
post #76

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

I agree. I think that the author may not have experienced these sorts of errors before. Yes, the average person may not care about experiencing a couple of bit flips per year and losing the odd pixel or block of a JPEG, but they will care if some cable somewhere or transfer or bad RAM chip or whatever else manages to destroy a significant amount of data before they notice it.

I had a significant data loss years ago.

I was young and only had a deskop, so all my data was there.

So I purchased a 300GB external usb drive to use for periodic backup. It was all manual copy/paste files across with no real schedule, but it was fine for the time and life was good.

Over time my data grew and the 300GB drive wasn't large enough to store it all. For a while some of it wasnt backed up (I was young with much less disposable income).

Eventually I purchased a 500GB drive.

But what I didn't know is my desktop drive was dying. Bits were flipping, a lot of them.

So when I did my first backup with the new drive I copied all my data off my desktop along with the corruption.

It was months before I realised a huge amount of my files were corrupted. By that point I'd wiped the old backup drive to give to my Mum to do her own backups. My data was long gone.

Once I discovered ZFS I jumped on it. It was the exact thing that would have prevented this because I could have detected the corruption when I purchased the new backup drive and did the initial backup to it.

(I made up the drive sizes because I can't remember, but the ratios will be about right).

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#122

I run a similar but less sophisticated setup. About 18 TiB now, and I run it 16 hours a day. I let it sleep 8 hours per night so that it's well rested in the morning. I just do this on a cron because I'm not clever enough to SSH into a turned off (and unplugged!) machine. 4 drives: 42k hours (4.7 years), 27k hours (3 years), 15k hours (1.6 years), and the last drive I don't know because apparently it isn't SMART. 0 e…

Looks like you're quite clever actually, if you can get cron to run on a powered off unplugged machine. I think I'm missing something.

Some bios and firmware support turning on at a certain time. Maybe cron was a way to simplify.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#123

Earlier quoted context omitted.

Drives are sealed anyway. Humidity maybe. Dust can’t really get in. Power or bad batch of drives.

Except for the helium-filled ones, they aren't sealed; there is a very fine filter that equalises atmospheric pressure. (This is also why they have a maximum operating altitude --- the head needs a certain amount of atmospheric pressure to float.)

Yup, this is why the label will say something along the lines of "DO NOT COVER DRIVE HOLES".

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#124
post #76

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

> Data integrity is the natural expectation humans have from computers I've said it once, and I'll say it again: the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist. No serious person designing a filesystem today would say it's okay to misplace your data. Not long ago, on this forum, someone told me that ZFS is only good because it had no competitors in its sp…

To paraphrase, "ZFS is the worst filesystem, except for all those other filesystems that have been tried from time to time."

It's far from perfect, but it has no peers.

I spent many years stubbornly using btrfs and lost data multiple times. Never once did the redundancy I had supposedly configured actually do anything to help me. ZFS has identified corruption caused by bad memory and a bad CPU and let me know immediately which files were damaged.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#127

Do you have a drive rotation schedule? 24 drives. Same model. Likely the same batch. Similar wear. Imagine most of them failing at the same time, and the rest failing as you're rebuilding it due to the increased load, because they're already almost at the same point. Reliable storage is tricky.

Reminds me of the time back in the day when Dell shipped us a server with drives serial numbers being consecutive.

Of course both failed at the same time and I spent an all nighter doing a restore.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#128
post #76

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

> Data integrity is the natural expectation humans have from computers I've said it once, and I'll say it again: the only reason ZFS isn't the norm is because we all once lived through a primordial era when it didn't exist. No serious person designing a filesystem today would say it's okay to misplace your data. Not long ago, on this forum, someone told me that ZFS is only good because it had no competitors in its sp…

The reason ZFS isn't the norm is because it historically was difficult to set up. Outside of NAS solutions, it's only since Ubuntu 20.04 it has been supported out of the box on any high profile customer facing OS. The reliability of the early versions was also questionable, with high zsys cpu usage and some times arcane commands needed to rebuild pools. Anecdotally, I've had to support lots of friends with zfs issues, never so with other file systems. The data always comes back, it's just that it needs petting.

Earlier, there used to be lot of fears around the license, with Torvalds advising against its use, both for that reason and for lack of maintainers. Now i believe that has been mostly ironed out and should be less of an issue.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#129
> Losing the system due to power shenanigans is a risk I accept.

There is another (very rare) failure an ups protects against, and that's imbalance in the electricity.

You can get a spike (up or down, both can be destructive) if there is construction in your area and something happens with the electricity, or lightning hits a pylon close enough to your house.

First job I worked at had multiple servers die like that, roughly 10 yrs ago. it's the only time I've ever heard of such an issue however

To my understanding, an ups protects from such spikes as well, as it will die before letting your servers get damaged

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#130
post #129

> Losing the system due to power shenanigans is a risk I accept. There is another (very rare) failure an ups protects against, and that's imbalance in the electricity. You can get a spike (up or down, both can be destructive) if there is construction in your area and something happens with the electricity, or lightning hits a pylon close enough to your house. First job I worked at had multiple servers die like that,…

Electronics is absolutely sensitive to this.

Please use filters.

Post reply on HN