ZFS: Use mirror vdevs, not RAIDZ • Hacker News Board

Re: ZFS: Use mirror vdevs, not RAIDZ

#51

Post by EvanAnderson » Wed, Dec 09, 2020, 4:59 PM UTC

metalliqaz wrote: post #39

dsr_ wrote: post #33

Earlier quoted context omitted.

zfs send. Keep different filesystems for system backup, personal data, media storage, gross hoarding. Consider not backing up the media storage and gross hoarding filesystems. Targets: 1. That NAS off in the corner built from an old desktop. 2. rsync.net and similar. 3. your encrypted cloud backup service of choice, through an adapter. If the value of your data is less than the cost to protect it, you know it is disp…

Yeah as much as my daughter likes the Trolls movie, I'm not paying to back that up.

Redacted zfs send/receive in OpenZFS 2.0 might be interesting to you: https://github.com/openzfs/zfs/pull/7958

Re: ZFS: Use mirror vdevs, not RAIDZ

#52

Post by aidenn0 » Wed, Dec 09, 2020, 5:00 PM UTC

rsync wrote: post #37

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable . We all sleep quite well with this arrangement and have…

I was running 3x 12 drive vdevs in raidz2 and write performance was terrible. We moved some data to a different machine and rebuilt as 18 mirrors. This was a long time ago (as in running on Solaris long time ago), so maybe things are better now.

Re: ZFS: Use mirror vdevs, not RAIDZ

#53

Post by gh02t » Wed, Dec 09, 2020, 5:02 PM UTC

minimaul wrote: post #13

minimaul wrote: post #8

As always with RAID-style setups, there’s an inevitable trade off of cost vs capacity vs performance. There’s still a place for RAIDZ/RAIDZ2, and in my opinion that place is storing bulk data that isn’t too heavily accessed or that needs to be stored with an eye towards keeping £/GB down. Yes, mirrors are faster. Yes, mirrors are easier to expand. But across 12 4TB disks that is 24TB instead of 40TB with RAIDZ2 - and…

The one thing in the article I agree with without reservation though is that you should always have a backup of your pool! ZFS makes doing good backups easy, with zfs send | zfs receive.

I mean, is this controversial? The problem for a budget conscious home user is: where do you back up to? Doing a complete backup to a remote storage provider isn't a practical option for many users, which leaves the affordable options being to only back up critical data and/or use something like a big USB external drive.

Neither of these options are great, so in reality many people end up relying on RAID or mirroring to provide some measure of redundancy regardless of whether or not it's a good idea.

Re: ZFS: Use mirror vdevs, not RAIDZ

#54

Post by louwrentius » Wed, Dec 09, 2020, 5:03 PM UTC

I strongly disagree with this old blogpost.

I feel that this advice is a somewhat dishonest attempt to plaster over the fact that you can't expand a VDEV.

https://louwrentius.com/the-hidden-cost-of-using-zfs-for-you...

So you try to burry that fact by promoting mirrors. But mirrors aren't as safe as RAIDZ2 and they aren't as space/efficient.

It all depends on circumstances, but if you want to store a ton of data, RAIDZ(2|3) seems the right way to go.

Use RAIDZ(2|3) vdevs, not mirrors.

Re: ZFS: Use mirror vdevs, not RAIDZ

#55

Post by metalliqaz » Wed, Dec 09, 2020, 5:15 PM UTC

EvanAnderson wrote: post #51

metalliqaz wrote: post #39

Earlier quoted context omitted.

Yeah as much as my daughter likes the Trolls movie, I'm not paying to back that up.

Redacted zfs send/receive in OpenZFS 2.0 might be interesting to you: https://github.com/openzfs/zfs/pull/7958

I already have selective backups to Backblaze with rclone.

The point is that home users can have good reasons for relying on redundancy.

Re: ZFS: Use mirror vdevs, not RAIDZ

#56

Post by seized » Wed, Dec 09, 2020, 5:17 PM UTC

rsync wrote: post #46

tutfbhuf wrote: post #42

Earlier quoted context omitted.

May I ask what your worst incident with ZFS was ever since 2012?

Some background ... We have a fairly robust drive-proofing procedure prior to deployment - I think we still use 'badblocks' and really beat them up for 5-7 days. It's not enough to merely not fail - we have to see perfect returns on the SMART diagnostics and zero complaints from FreeBSD. So we're starting with a known-good population of drives. Then we monitor them very closely - again with both SMART and FreeBSD/ZFS…

Agreed on the limping along. I survived a RAIDZ1 double drive failure exactly like that, one drive completely died and a second started acting up during the resilver. Resilver slowed way down but made it eventually.

All those drives got replaced with a RAIDZ2 pool.

Re: ZFS: Use mirror vdevs, not RAIDZ

#57

Post by mulmen » Wed, Dec 09, 2020, 5:17 PM UTC

tjoff wrote: post #22

I don't agree. And if one of your disks failed, and age was a factor… you’re going to be sweating bullets wondering if another will fail before your resilver completes. So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool. I sure do hope you aren't on vacation and/or have to order a drive online. It all depends on your use case. For me, raidz3 wins ea…

What’s wrong with rsync.net or tarsnap for backups?

Re: ZFS: Use mirror vdevs, not RAIDZ

#58

Post by mnd999 » Wed, Dec 09, 2020, 5:18 PM UTC

dsr_ wrote: post #33

tjoff wrote: post #25

Earlier quoted context omitted.

RAID is not a backup solution, its an uptime solution. I know. Doesn't change the fact there exist no sensible way to backup a decently sized pool in a home-user scenario. Now we might disagree on what is sensible. And some things might not need a backup.

zfs send. Keep different filesystems for system backup, personal data, media storage, gross hoarding. Consider not backing up the media storage and gross hoarding filesystems. Targets: 1. That NAS off in the corner built from an old desktop. 2. rsync.net and similar. 3. your encrypted cloud backup service of choice, through an adapter. If the value of your data is less than the cost to protect it, you know it is disp…

2) and 3) requires a decent broadband upload speed. Home broadband often doesn’t provide this.

Re: ZFS: Use mirror vdevs, not RAIDZ

#59

Post by kaliszad » Wed, Dec 09, 2020, 5:21 PM UTC

tjoff wrote: post #25

necheffa wrote: post #24

Earlier quoted context omitted.

Repeat after me: RAID is not a backup solution, its an uptime solution.

RAID is not a backup solution, its an uptime solution. I know. Doesn't change the fact there exist no sensible way to backup a decently sized pool in a home-user scenario. Now we might disagree on what is sensible. And some things might not need a backup.

Some people mention ZFS send feature, which should prevent sending unchanged blocks. You could use rsnapshot to a different machine (that needs to be just a little bit larger than the original one) and personally I would recommend borg backup, which can deduplicate, compress and encrypt and isn't hard to setup. This could result in the backups being a fraction of the original - depending on the data. You can use AWS S3/ DO Spaces which is a lot cheaper or other compatible interfaces using rclone serve sftp + sshfs with borg or in the case of restic or duplicacy natively. You can also lookup tarsnap, rsync.net and others for remote backups.

In home scenarios, you usually care about documents, pictures a few specific videos and some music above everything else. This tends to be a fraction of the total capacity. The rest doesn't change very often and is not such a loss but tends to be quite huge in comparison. So to backup all documents and pictures one or two external drives and/ or a cloud backup is sufficient.

Personally, for some important documents I use orgpad.com which I help build. I am personally responsible for the infrastructure - so the backups actually have at least three tiers, independent providers etc. That is just a side effect of having a SaaS platform with a reasonable and contactable team behind it.

Re: ZFS: Use mirror vdevs, not RAIDZ

#60

Post by sigstoat » Wed, Dec 09, 2020, 5:22 PM UTC

tjoff wrote: post #22

I don't agree. And if one of your disks failed, and age was a factor… you’re going to be sweating bullets wondering if another will fail before your resilver completes. So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool. I sure do hope you aren't on vacation and/or have to order a drive online. It all depends on your use case. For me, raidz3 wins ea…

> Except for maybe another pool. Which is going to hurt since ZFS with buying everything up front is VASTLY more expensive than RAID where you can grow the array as needed - under the assumption that the storage needs grow slowly, which they typically do for home users.

your backup pool doesn't have to match your working pool. it certainly doesn't have to be the same size. it just has to be larger than the data you've got.

so you can build a big 64TB pool, and back it up to a set of external USB drives which expand as your actual data expands. similarly you can turn on a bit more compression for the backup pool than you might want on the main one.