Live data from Hacker News

ZFS: Use mirror vdevs, not RAIDZ

jrs-s.net

21–30 of 133 posts

Re: ZFS: Use mirror vdevs, not RAIDZ

#21
post #8

As always with RAID-style setups, there’s an inevitable trade off of cost vs capacity vs performance. There’s still a place for RAIDZ/RAIDZ2, and in my opinion that place is storing bulk data that isn’t too heavily accessed or that needs to be stored with an eye towards keeping £/GB down. Yes, mirrors are faster. Yes, mirrors are easier to expand. But across 12 4TB disks that is 24TB instead of 40TB with RAIDZ2 - and…

> The rebuild times in this post seem high to me, though.

Two factors. Disks are getting large, and the rebuild time for RAID-Z[1] is dependent on the fragmentation. In combination it means it can take ages.

I just had to replace a failing WD Red 3TB[2] in an old 4xRAID-Z1 pool, and it took 9 hours. That was a single 3TB disk. The disks in my new pool are 14TB and 16TB.

[1]: https://youtu.be/Efl0Kv_hXwY

[2]: power-on hours in SMART showed over 7 years

Re: ZFS: Use mirror vdevs, not RAIDZ

#22
I don't agree.

And if one of your disks failed, and age was a factor… you’re going to be sweating bullets wondering if another will fail before your resilver completes.

So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool. I sure do hope you aren't on vacation and/or have to order a drive online.

It all depends on your use case. For me, raidz3 wins easily. Performance during resilvering is not something most users would suffer that much from anyway either. And that 8-drive recommendation comes from many factors and don't apply to most home-users anyway. Not that you should go overboard with it but it will easily pay for fast cache or whatever else you might want.

If performance was the goal you would not be using spinning rust anyway, and if you are still limited by a single gbit then don't even think about it (but don't over-utilize your pool).

But there are still lots of potential ways for your data to die, and you still need to back up your pool. Period. PERIOD!

Of course! Yet for home users there is not a single reasonable way to do it if you have decently sized pool. There is bound to be sacrifices on what you choose to backup.

Except for maybe another pool. Which is going to hurt since ZFS with buying everything up front is VASTLY more expensive than RAID where you can grow the array as needed - under the assumption that the storage needs grow slowly, which they typically do for home users.

As a ZFS user and fanboy the temptation of a dual raid6 setup is tempting. But I can't compromise on the filesystem so in the end I've compromised on backups instead. Likely not the smartest move considering how rare bitrot is, I am very well aware.

For many drives I'd go for raidz3. And for a 4-drive NAS I'd go with raidz2 rather than a mirror setup for the ease of mind.

For fast SSD pools I'd go with mirrors. Much easier to backup the entire pool as well.

Re: ZFS: Use mirror vdevs, not RAIDZ

#23
post #8

As always with RAID-style setups, there’s an inevitable trade off of cost vs capacity vs performance. There’s still a place for RAIDZ/RAIDZ2, and in my opinion that place is storing bulk data that isn’t too heavily accessed or that needs to be stored with an eye towards keeping £/GB down. Yes, mirrors are faster. Yes, mirrors are easier to expand. But across 12 4TB disks that is 24TB instead of 40TB with RAIDZ2 - and…

> The rebuild times in this post seem high to me, though. Two factors. Disks are getting large, and the rebuild time for RAID-Z[1] is dependent on the fragmentation. In combination it means it can take ages. I just had to replace a failing WD Red 3TB[2] in an old 4xRAID-Z1 pool, and it took 9 hours. That was a single 3TB disk. The disks in my new pool are 14TB and 16TB. [1]: https://youtu.be/Efl0Kv_hXwY [2]: power-on…

OpenZFS 2.0 has sequential rebuilds:

> The sequential reconstruction feature adds a more traditional RAID rebuild mechanism to ZFS. Specifically, it allows for mirror vdevs to be rebuilt in LBA order. Depending on the pools average block size, overall fragmentation, and the performance characteristics of the devices (SMR) sequential reconstruction can restore redundancy in less time than a traditional healing resilver. However, it cannot verify block checksums as part of the rebuild. Therefore a scrub is automatically started when the last active sequential resilver completes.

* https://github.com/openzfs/zfs/pull/10349

Re: ZFS: Use mirror vdevs, not RAIDZ

#24
post #22

I don't agree. And if one of your disks failed, and age was a factor… you’re going to be sweating bullets wondering if another will fail before your resilver completes. So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool. I sure do hope you aren't on vacation and/or have to order a drive online. It all depends on your use case. For me, raidz3 wins ea…

Repeat after me: RAID is not a backup solution, its an uptime solution.

Re: ZFS: Use mirror vdevs, not RAIDZ

#25
post #22

I don't agree. And if one of your disks failed, and age was a factor… you’re going to be sweating bullets wondering if another will fail before your resilver completes. So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool. I sure do hope you aren't on vacation and/or have to order a drive online. It all depends on your use case. For me, raidz3 wins ea…

Repeat after me: RAID is not a backup solution, its an uptime solution.

RAID is not a backup solution, its an uptime solution.

I know. Doesn't change the fact there exist no sensible way to backup a decently sized pool in a home-user scenario.

Now we might disagree on what is sensible. And some things might not need a backup.

Re: ZFS: Use mirror vdevs, not RAIDZ

#26
There's no planet on which I would trust a mirror with a 10TB + SATA drive. There's a reason the major storage vendors already have or are working on 3-disk parity for large NL-SAS/SATAS drives.

Give me RAID-Z2 or RAID-Z3 with dRAID all day long (although I wouldn't deploy dRAID quite yet on production workloads).

Re: ZFS: Use mirror vdevs, not RAIDZ

#27
I would like to see an update for this for SSDs. The devices are black boxes of mystery, possibly likely to fail nearly simultaneously (at least in terms of being unable to write). Early on we left 20% if capacity unformatted to allow more wear leveling, but running them in RAID1 seemed both necessary and likely to cause synchronous errors. RAIDZ1 over three disks might be okay. Obviously huge R/W bandwidth so resilvering doesn't take long.

Also, it isn't mentioned, but having the ZIL on battery backed flash can give huge improvements in IOPS to anything, and is far more valuable than extra TB or spindles.

Re: ZFS: Use mirror vdevs, not RAIDZ

#28

Earlier quoted context omitted.

> The rebuild times in this post seem high to me, though. Two factors. Disks are getting large, and the rebuild time for RAID-Z[1] is dependent on the fragmentation. In combination it means it can take ages. I just had to replace a failing WD Red 3TB[2] in an old 4xRAID-Z1 pool, and it took 9 hours. That was a single 3TB disk. The disks in my new pool are 14TB and 16TB. [1]: https://youtu.be/Efl0Kv_hXwY [2]: power-on…

OpenZFS 2.0 has sequential rebuilds: > The sequential reconstruction feature adds a more traditional RAID rebuild mechanism to ZFS. Specifically, it allows for mirror vdevs to be rebuilt in LBA order. Depending on the pools average block size, overall fragmentation, and the performance characteristics of the devices (SMR) sequential reconstruction can restore redundancy in less time than a traditional healing resilve…

Yeah I linked to the OpenZFS Developer Summit video on the feature, but forgot to mention it in my post.

Re: ZFS: Use mirror vdevs, not RAIDZ

#29
I looked for, and didn't see, "SSD" in this article (let alone "NVMe"). Maybe because it's from 2015? But at any rate, I'm not sure the logic applies there. High performance SSDs remain much more expensive, so losing major capacity is a much costlier issue, and simultaneously they rebuild vastly faster. I thought about this when making a pool out of U.2 NVMe drives, and with rebuild times measured in minutes and given the cost/GB I think RAIDZ2 (or even Z1) vdevs are plenty sufficient for most use cases.

By the same token, what does the backup system and unique pool data lifetime look like? If someone is using a very fast/smaller/expensive pool as a local working space, but it's constantly being replicated to a much more heavily redundant pool of spinning rust in turn backing up sufficiently fast to remote, it may be perfectly acceptable to have minimal redundancy (I still like being able to heal from corruption) in the working pool. If the whole thing going kaput only means losing a few minutes of data it's totally reasonable to consider how much money that's actually worth.

I guess a lot of the blanket advice for ZFS rubs me the wrong way. It's offers a very powerful toolbox full of options that are genuinely great in different circumstances, and there aren't many footguns (dedup being the biggest one that immediately comes to mind) that are hard to reason about. It's a shame if users aren't considering there own budgets, needs, hardware, and so on and taking advantage of it to get the most out of them.

Re: ZFS: Use mirror vdevs, not RAIDZ

#30
post #25

Earlier quoted context omitted.

Repeat after me: RAID is not a backup solution, its an uptime solution.

RAID is not a backup solution, its an uptime solution. I know. Doesn't change the fact there exist no sensible way to backup a decently sized pool in a home-user scenario. Now we might disagree on what is sensible. And some things might not need a backup.

Isn’t the backup solution to always get two of whatever you build? A primary raid backed up to a second pool?
Post reply on HN