Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS – add disks to existing RAIDZ

github.com

Post reply on HN
Print view View original

151–160 of 177 posts

Re: OpenZFS – add disks to existing RAIDZ

#151

Post by crote »

mustache_kimono wrote: post #42
crote wrote: post #29

> After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks. New blocks will be written with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been expanded once to 6-wide, has 4 data to 2 parity). Does anyone know why this is the case? When expanding an array which is getting full this wi…

> Does anyone know why this is the case? > Considering the process is already reading and rewriting basically the entire array, why not recalculate the parity as well? IANA expert but my guess is -- because, here, you don't have to modify block pointers, etc. ZFS RAIDZ is not like traditional RAID, as it's not just a sequence of arbitrary bits, data plus parity. RAIDZ stripe width is variable/dynamic, written in bloc…

But you are moving the blocks to other disks anyway, surely that would require a change of the block pointers too?

Re: OpenZFS – add disks to existing RAIDZ

#152

Post by crote »

allanjude wrote: post #65
crote wrote: post #29

> After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks. New blocks will be written with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been expanded once to 6-wide, has 4 data to 2 parity). Does anyone know why this is the case? When expanding an array which is getting full this wi…

That is not how this will work. The reason the parity ratio stays the same, is that all of the references to the data are by DVA (Data Virtual Address, effectively the LBA within the RAID-Z vdev). So the data will occupy the same amount of space and parity as it did before. All stripes in RAID-Z are dynamic, so if your stripe is 5 wide and your array is 6 wide, the 2nd stripe will start on the last disk and wrap arou…

The linked pull request says "After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks". That'd mean that the disks do not contain the same data, but it is getting moved around?

Regardless, my entire point is that you still lose a significant amount of capacity due to the old data remaining as 3+2 rather than being rewritten to 4+2, which heavily disincentives the expansion of arrays reaching capacity - but that is the only time people would want to expand their array.

It just seems to me like they are spending a lot of effort on a feature which you frankly should not ever want to use.

Re: OpenZFS – add disks to existing RAIDZ

#153

Post by crote »

magicalhippo wrote: post #46
crote wrote: post #29

> After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks. New blocks will be written with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been expanded once to 6-wide, has 4 data to 2 parity). Does anyone know why this is the case? When expanding an array which is getting full this wi…

> Considering the process is already reading and rewriting basically the entire array, why not recalculate the parity as well? Because snapshots might refer to the old blocks. Sure you could recompute, but then any snapshots would mean those old blocks would have to stay around so now you've taken up ~twice the space.

Or you could just rewrite the snapshots while you are at it.

Re: OpenZFS – add disks to existing RAIDZ

#154

Post by labawi »

ErneX wrote: post #118
edmundsauto wrote: post #73

Earlier quoted context omitted.

Wouldn't it be better to apply stress over more drives to minimize the chances that you lose a second drive during a rebuild?

I believe this is the article I read when I started: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...

Note that the listed probabilities of surviving N-disk failures are not correct, though after calculation, though the differences may not be that important.

Listed survival probability of N-disk failure in 8-drive/4-vdev mirror.

1: 1; 2: 0.857; 3: 0.667; 4: 0.400; 5: 0; 6: N/A; 7: N/A; 8: N/A

Proper survival probability:

1: 1; 2: 0.857; 3: 0.571; 4: 0.229; 5: 0; 6: 0; 7: 0; 8: 0

Comparatively, survival probability for 8-drive RAIDz4 with equal amount of usable space:

1: 1; 2: 1; 3: 1; 4: 1; 5: 0; 6: 0; 7: 0; 8: 0

Personally, I'd use multi-vdev mirror pools only for data that is either backed-up or data I can afford to lose completely.

Re: OpenZFS – add disks to existing RAIDZ

#155

Post by rincebrain »

gliaskos wrote: post #122
ksec wrote: post #100

Earlier quoted context omitted.

If you follow Qnap you may be aware of their security nightmare in recent years. And HERO OS is only available on the Enterprise range of NAS, not even Prosumer class NAS.

Something like TVS-h474 seems quite nice for a wide range of uses. I have TVS-h1288X and I am really happy with it, maybe it's a bit overkill for most. That said, I could not use zfs send from my prev fbsd server, there are some compat checks in QNAP's version. Used nfs to migrate from my decade++ old pool. Using Tailscale for access, they have QNAP builds on GH.

QNAP forked from OpenZFS a long time ago, before the ZoL project was renamed the canonical OpenZFS project IIRC, and has been reimplementing whatever they wanted for some time.

So it's not just a check for check's sake, I would suspect in practice their send/recv is just completely incompatible at this point.

Re: OpenZFS – add disks to existing RAIDZ

#156

Post by rincebrain »

lloeki wrote: post #114
wkat4242 wrote: post #87

Earlier quoted context omitted.

Could you force a complete rewrite if you wanted to? That would be handy. Without copying all the data elsewhere of course. I don't have another 90TB of spare disks :P Edit: I suppose I could cover this with a shell script needing only the spare space of the largest file. Nice!

> Could you force a complete rewrite if you wanted to? On btrfs that's a rebalance, and part of how one expands an array (btrfs add + btrs balance) (Not sure if ZFS has a similar operation, but from my understanding resilvering would not be it) Not that it matters much though as RAID5 and RAID6 aren't dependable upon, and the array failure modes are weird in practice, so in context of expanding storage it really only…

ZFS does not, and fundamental is not going to ever get one without rewriting so much you'd cry.

Re: OpenZFS – add disks to existing RAIDZ

#157

Post by rincebrain »

crote wrote: post #152
allanjude wrote: post #65

Earlier quoted context omitted.

That is not how this will work. The reason the parity ratio stays the same, is that all of the references to the data are by DVA (Data Virtual Address, effectively the LBA within the RAID-Z vdev). So the data will occupy the same amount of space and parity as it did before. All stripes in RAID-Z are dynamic, so if your stripe is 5 wide and your array is 6 wide, the 2nd stripe will start on the last disk and wrap arou…

The linked pull request says "After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks". That'd mean that the disks do not contain the same data, but it is getting moved around? Regardless, my entire point is that you still lose a significant amount of capacity due to the old data remaining as 3+2 r…

I don't think that's true.

I don't use raidz for my personal pools because it has the wrong set of tradeoffs for my usage, but if I did, I'd absolutely use this.

Yes, your data has the old data:parity ratio for older data, but you now have more total storage available, which is the entire goal. Sure, it'd be more space-efficient to go rewrite your data, piecemeal or entirely, afterward, but you now have more storage to work with, rather than having to remake the pool or replace every disk in the vdev with a larger one.

Re: OpenZFS – add disks to existing RAIDZ

#158

Post by rincebrain »

didntcheck wrote: post #123
katbyte wrote: post #69

Earlier quoted context omitted.

I imagine there would eventually be a way / option to automatically rewrite old blocks. Or at least I would hope so because usually when your adding new disks the array is going to be near full

IIRC one of the developers was asked about this and said it would be about as much additional effort as the whole expansion feature so far. There is of course the user-level workaround of (roughly) `for file in *; do cp $file $file.new && mv $file.new $file && done`, but if I understand correctly you'd need to do this with no snapshots, otherwise the space from $file wouldn't be freed so you'd need an array that's at…

It'd probably be worse than that.

I cannot stress how expensive and invasive that would be enough.

Re: OpenZFS – add disks to existing RAIDZ

#159

Post by tatersolid »

wtallis wrote: post #82
ilyt wrote: post #76

Earlier quoted context omitted.

Eh, it can kinda work. If you have 1, 2,3,4TB drives you can make RAID10 out of first GB, then stitch 2TB RAID0 from second and third and RAID1 it with 2TB out of 4th drive. then raid1 out of remaining GB on 3rd and 4th. Then glue it together with LVM and hope for best.

Or you can just put all those devices into one BTRFS RAID 1 filesystem and not have to worry about it.

Until the BTRS filesystem gets to some indeterminate threshold of “almost full”, at which point you typically need to reformat and restore from backups…

Re: OpenZFS – add disks to existing RAIDZ

#160

Post by wtallis »

tatersolid wrote: post #159
wtallis wrote: post #82

Earlier quoted context omitted.

Or you can just put all those devices into one BTRFS RAID 1 filesystem and not have to worry about it.

Until the BTRS filesystem gets to some indeterminate threshold of “almost full”, at which point you typically need to reformat and restore from backups…

I haven't been following BTRFS development closely for a few years, so I don't know how much more progress they've made on eliminating those corner cases. But I'm still careful about not letting the filesystem get close to full. Earlier this month, I noticed one of my machines was over 80% full, so I grabbed a few drives I had lying around that weren't in use at the time and added them to the array. The flexibility of BTRFS gives you a lot more options than eg. ZFS for getting out of a tight spot without the full wipe and restore hassle. But both can still leave you stuck, especially if you ignore the need for monitoring and preventive maintenance.
Post reply on HN