Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

louwrentius.com

Post reply on HN
Print view View original

191–200 of 314 posts

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#191

Post by lazide »

willis936 wrote: post #176
369548684892826 wrote: post #170

Earlier quoted context omitted.

There’s something disturbing about the idea of silent data loss, it totally undermines the peace of mind of having backups. ZFS is good, but you can also just run rsync periodically with checksum and dryrun args and check the output for diffs.

It happens all the time. Have a plan, perform fire drills. It's a lot of time and money, but there's no equivalent feeling to unfucking yourself quite like being able to get your lost, fragile data back.

The challenge with silent data loss is your backups will eventually not have the data either - it will just be gone, silently.

After having that happen a few times (pre-ZFS), I started running periodic find | md5sum > log.txt type jobs and keeping archives.

It’s caught more than a few problems over the years, and allows manual double checking even when using things like ZFS. In particular, some tools/settings just aren’t sane to use to copy large data sets, and I only discovered that when… some of it didn’t make it to it’s destination.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#192

Post by nolok »

mrjin wrote: post #188
nolok wrote: post #147

Earlier quoted context omitted.

That's all fine and good until that one random lone broken block stops you from opening that file you really need.

What you need is backup. RAID is not backup and is not for most home/personal users. I learnt that the hard way. Now my NAS use simple volumes only, after all, I really don't have many things I cannot lose on it. If it's something really important, I have multiple copies on different drives, and some offline cold backup. So now if any of my NAS drive is about to fail, I can just copy out the data and replace the driv…

Oh I have backups myself. But parent is more or less talking about a 71TiB NAS for residential usage and being able to ignore the bit rot; in that context such a person probably wouldn't have backup.

Personnaly I have long since moved out of raid 5/6 into raid 1 or 10 with versionned backup, at some level of data raid 5/6 just isn't cutting it anymore in case anything goes slightly wrong.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#193

Post by wazoox »

I currently support many NAS servers in the 50TB - 2PB range, many of them being 10, 12, and up to 15 years old for some of them. Most of them still run with their original power supplies, motherboard and most of their original (HGST -- now WD -- UlstraStar) drives, though of course a few drives have failed for some of them (but not all).

2, 4, 8TB HGST UltraStar disks are particularly reliable. All of my desktop PCs currently hosts mirrors of 2009 vintage, 2 TB drives that I got when they're put out of service. I have heaps of spare, good 2 TB drives (and a few hundreds still running in production after all these years).

For some reason 14TB drives seem to have a much higher failure rate than Helium drives of all sizes. On a fleet of only about 40 14 TB drives, I had more failures than on a fleet of over 1000 12 and 16 TB.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#194

Post by AndrewDavis »

369548684892826 wrote: post #170
AndrewDavis wrote: post #121

Earlier quoted context omitted.

I had a significant data loss years ago. I was young and only had a deskop, so all my data was there. So I purchased a 300GB external usb drive to use for periodic backup. It was all manual copy/paste files across with no real schedule, but it was fine for the time and life was good. Over time my data grew and the 300GB drive wasn't large enough to store it all. For a while some of it wasnt backed up (I was young wit…

There’s something disturbing about the idea of silent data loss, it totally undermines the peace of mind of having backups. ZFS is good, but you can also just run rsync periodically with checksum and dryrun args and check the output for diffs.

Absolutely, if you can't use a filesystem with checksums (zfs, btrfs, bcachefs) then rsync is a great idea.

I think filesystem checksums have one big advantage vs rsync. With rsync if there's a difference it isn't clear which one is wrong.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#195

Post by homebrewer »

KMag wrote: post #167
doublepg23 wrote: post #152

Earlier quoted context omitted.

That was always FUD more or less. ZFS uses RAM as its primary cache…like every other filesystem, so it if you have very little RAM for caching the performance will degrade…like every other filesystem.

But if you have a single board computer with 1 GB of RAM and several TB of ZFS, will it just be slow, or actually not run? Granted, my use case was abnormal, and I was evaluating in the early days when there were both license and quality concerns with ZFS on Linux. However, my understanding at the time was that it wouldn't actually work to have several TB in a ZFS pool with 1 GB of RAM. My understanding is that ZFS h…

> will it just be slow

This. I use it on a tiny backup server with only 1 GB of RAM and a 4 TB HDD pool, it's fine. Only one machine backs up to that server at a time, and they do that at network speed (which is admittedly only 100 Mb/s, but it should go somewhat higher if it had faster network). Restore also runs ok.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#197

Post by rollcat »

snvzz wrote: post #76

>but for residential usage, it's totally reasonable to accept the risk. Polite disagree. Data integrity is the natural expectation humans have from computers, and thus we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

> we should stick to filesystems with data checksums such as ZFS, as well as ECC memory.

While I don't disagree with this statement, consider the reality:

- APFS has metadata checksums, but no data checksums. WTF Apple?

- Very few Linux distributions ship zfs.ko (&spl.ko); those that do, theoretically face a legal risk (any kernel contributor could sue them for breaching the GPL); rebuilding the driver from source is awkward (even with e.g. DKMS), pulls more power, takes time, and may randomly leave your system unbootable (YES it happened to me once).

- Linux itself explicitly treats ZFS as unsupported; loading the module taints the kernel.

- FreeBSD is great, and is actually making great progress catching up with Linux on the desktop. Still, it is a catch-up game. I also don't want to install a system that needs to install another guest system to actually run the programs I need.

- There are no practical alternatives to ZFS that even come close; sibling comment complains about btrfs data loss. I never had the guts to try btrfs in production after all the horror stories I've heard over the decade+.

- ECC memory on laptops is practically unheard of, save for a couple niche Thinkpad models; and comes with large premiums on desktops.

What are the practical choices for people who do not want to cosplay as sysadmins?

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#198

Post by hulitu »

lazide wrote: post #175
pbhjpbhj wrote: post #164

Earlier quoted context omitted.

I only know about FAT but these "key file metadata blocks" are redundant, so you need really special double-plus bad luck to do that.

It was ext4, and I’ve had it happen two different times - in fact, I’ve never had it happen in a ‘good’ recoverable way before that I’ve ever seen. It triggered a kernel panic in every machine that I mounted it in, and it wasn’t a media issue either. Doing a block level read of the media had zero issues and consistently returned the exact same data the 10 times I did it. Notably, I had the same thing happen using btr…

Good to know. Ext2 is much more robust against corruption. Or at least it was 10 years ago when i had kernel crashes or power failures.

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#199

Post by [deleted] »

BSDobelix wrote: post #187
doublepg23 wrote: post #152

Earlier quoted context omitted.

That was always FUD more or less. ZFS uses RAM as its primary cache…like every other filesystem, so it if you have very little RAM for caching the performance will degrade…like every other filesystem.

>That was always FUD more or less Thank you thank you, exactly this! And additionally that cache is compressed. In the day's of 4GB machines ZFS was overkill but today...no problem.

[deleted]

Re: My 71 TiB ZFS NAS After 10 Years and Zero Drive Failures

#200

Post by londons_explore »

orbital-decay wrote: post #111

Do you have a drive rotation schedule? 24 drives. Same model. Likely the same batch. Similar wear. Imagine most of them failing at the same time, and the rest failing as you're rebuilding it due to the increased load, because they're already almost at the same point. Reliable storage is tricky.

Software bugs might cause that (eg. drive fails after exactly 1 billion IOPS due to some counter overflowing). But hardware wear probably won't be as consistent.
Post reply on HN