ZFS: Use mirror vdevs, not RAIDZ

May I ask what your worst incident with ZFS was ever since 2012?

False. There are too many different reasons for a backup to make that statement, and for most people RAID and NAS are combined into one.

There is backup because a disks are unreliable and fail randomly. RAID is designed to solve this.

There is backup because you accidentally deleted a file. There is backup because ransomware encrypted your files. RAID with snapshots solve these (arguably better than most other backup systems).

There is backup because your main computers (phone) might break. NAS systems are a good place to backup these.

There is backup because your house burned down [or otherwise is destroyed]. This is the only one that RAID doesn't solve. (even then you can do offsite NAS)

What risks do you need to mitigate? Once you have that list you can decide what you mitigations to apply. RAID is a very useful piece of this and so saying RAID is not backup is not helpful and may harm people from making useful backups because they let the search for perfection delay them.

Some background ...

We have a fairly robust drive-proofing procedure prior to deployment - I think we still use 'badblocks' and really beat them up for 5-7 days. It's not enough to merely not fail - we have to see perfect returns on the SMART diagnostics and zero complaints from FreeBSD.

So we're starting with a known-good population of drives.

Then we monitor them very closely - again with both SMART and FreeBSD/ZFS system data and we are fairly aggressive about failing them out early if we see them start to misbehave.

So we don't see things like straight up drive failures - the bad ones have already been rejected and drives that misbehave are culled out.

The last time I got pulled in to make a judgement call (and got a little nervous) was (IIRC) we had a 15 drive vdev that had a drive failure and a candidate for early removal that had started to misbehave and the choice was made to yank them both because why do two big long resilvers ... and then during the subsequent resilver a third drive, while not failing, started to spit out errors. So the resilver completed and all was well but we had a potential third failure that could have died in the resilver and then we would be running with no protection.

But that's why I love raidz3 - even if that had happened, we would have needed to lose yet another drive to lose the vdev (and the entire pool).

I think the actionable recommendations here are to burn in your drives when you get them because we do, indeed, find rejects in most batches. Also, pick an error threshold that is low and be disciplined about sticking to it - don't let drives spin out SMART errors sporadically for months ...

EDIT: Here is another thought and this goes back to pre-ZFS days and old-style RAID, etc. In normal operations we're aggressively failing out drives that misbehave BUT if you're in a marginal situation you need to flip that logic around - especially if the pool is in-use during resilver/rebuild.

If you've lost 2 drives in a raidz3 (or, say, 1 drive in a RAID6) that remaining protection drive even if it is failing is like gold. If you fail it out, it's gone forever and has no relevance to the pool/array - but if you keep it, even as it's dying, you can either limp along OR you can even offline the pool/array and send that last drive to recovery or clone it or whatever ... the point is, when an array goes sideways every single bit of parity, no matter how poorly behaved, should be treated like gold.

People act like resilvers on parity are vastly different than on mirrored. They are not, its just reads and writes, granted in a different pattern. My RAIDZ2 resilvers at about what I expect the old drives in it to write at, ~100MB/sec. In a mirrored setup it all depends on the one drive doing the reading to survive the process. The drive that is most likely the same age as the one that just failed.

A resilver isnt more likely to kill an existing drive than a scrub. And the general advice is to do those semi regularly.

A blanket statement equating parity RAID to "more risky" is absurd and leads to people thinking they need a massive pool of mirrors just for their home files. ZFS has numerous options to prioritize different IO higher or lower (priority to resilvers or end user data).

UFS2? So you are using FreeBSD?! Yet another reason to support you folks, seriously.