Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS – add disks to existing RAIDZ

github.com

Post reply on HN
Print view View original

161–170 of 177 posts

Re: OpenZFS – add disks to existing RAIDZ

#161

Post by magicalhippo »

crote wrote: post #153
magicalhippo wrote: post #46

Earlier quoted context omitted.

> Considering the process is already reading and rewriting basically the entire array, why not recalculate the parity as well? Because snapshots might refer to the old blocks. Sure you could recompute, but then any snapshots would mean those old blocks would have to stay around so now you've taken up ~twice the space.

Or you could just rewrite the snapshots while you are at it.

How would you go about doing that?

AFAIK in ZFS snapshots are just a pointer to (an old) merkle tree[1], if you go about changing the blocks then you need to update that tree, but you can't due to copy-on-write (without paying for the copy, like I mentioned).

[1]: https://openzfs.readthedocs.io/en/latest/introduction.html#d...

Re: OpenZFS – add disks to existing RAIDZ

#162

Post by tatersolid »

wtallis wrote: post #160
tatersolid wrote: post #159

Earlier quoted context omitted.

Until the BTRS filesystem gets to some indeterminate threshold of “almost full”, at which point you typically need to reformat and restore from backups…

I haven't been following BTRFS development closely for a few years, so I don't know how much more progress they've made on eliminating those corner cases. But I'm still careful about not letting the filesystem get close to full. Earlier this month, I noticed one of my machines was over 80% full, so I grabbed a few drives I had lying around that weren't in use at the time and added them to the array. The flexibility o…

Disk full is not a “corner case”. It should be one of the first test cases a filesystem test suite should have, and that test should always pass in every configuration.

The fact that total data loss happened regularly on full volumes many years after a “stable” release means BTRFS is either fundamentally flawed in design or incompetently implemented. I have no idea how it was accepted into mainline Linux.

The fact that a certain large NAS vendor used BTRFS by default (without documentation at the time of purchase) has cost my org and others lots of downtime and money.

Re: OpenZFS – add disks to existing RAIDZ

#163

Post by mustache_kimono »

crote wrote: post #151
mustache_kimono wrote: post #42

Earlier quoted context omitted.

> Does anyone know why this is the case? > Considering the process is already reading and rewriting basically the entire array, why not recalculate the parity as well? IANA expert but my guess is -- because, here, you don't have to modify block pointers, etc. ZFS RAIDZ is not like traditional RAID, as it's not just a sequence of arbitrary bits, data plus parity. RAIDZ stripe width is variable/dynamic, written in bloc…

But you are moving the blocks to other disks anyway, surely that would require a change of the block pointers too?

Again not an expert -- but if you look at the presentations re: this feature, I presume you wouldn't. Why? Because another feature, adopted concurrently, provides for time independent geometry? So, if you look at the pool after time X, you have the same block #, etc., but with a different geo.

Re: OpenZFS – add disks to existing RAIDZ

#164

Post by wtallis »

tatersolid wrote: post #162
wtallis wrote: post #160

Earlier quoted context omitted.

I haven't been following BTRFS development closely for a few years, so I don't know how much more progress they've made on eliminating those corner cases. But I'm still careful about not letting the filesystem get close to full. Earlier this month, I noticed one of my machines was over 80% full, so I grabbed a few drives I had lying around that weren't in use at the time and added them to the array. The flexibility o…

Disk full is not a “corner case”. It should be one of the first test cases a filesystem test suite should have, and that test should always pass in every configuration. The fact that total data loss happened regularly on full volumes many years after a “stable” release means BTRFS is either fundamentally flawed in design or incompetently implemented. I have no idea how it was accepted into mainline Linux. The fact th…

> Disk full is not a “corner case”.

I didn't say it was. There are plenty of disk full scenarios that can be handled by BTRFS without trouble, especially when adding more disks (even temporarily) is an option. There are ways to get stuck with a full fs, but it's not a guarantee you'll end up in a "wipe and restore from backup" situation. (And to be fair, ZFS also gets very problematic when completely full; this is a general problem for CoW filesystems.)

Re: OpenZFS – add disks to existing RAIDZ

#165

Post by ErneX »

ilyt wrote: post #145
ErneX wrote: post #118

Earlier quoted context omitted.

I believe this is the article I read when I started: https://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs...

That talks more about RAID5/Z1 vs mirroring. You still need 6/Z2 if you want to have reasonable fault tolerance, unless you want to waste ungodly amount of space.

My reasoning for using mirrored vdevs was more about the ease of expanding the storage than the redundancy.

Re: OpenZFS – add disks to existing RAIDZ

#166

Post by ilyt »

wtallis wrote: post #160
tatersolid wrote: post #159

Earlier quoted context omitted.

Until the BTRS filesystem gets to some indeterminate threshold of “almost full”, at which point you typically need to reformat and restore from backups…

I haven't been following BTRFS development closely for a few years, so I don't know how much more progress they've made on eliminating those corner cases. But I'm still careful about not letting the filesystem get close to full. Earlier this month, I noticed one of my machines was over 80% full, so I grabbed a few drives I had lying around that weren't in use at the time and added them to the array. The flexibility o…

Sooo basically if you were using XFS you wouldn't need to do anything at all till it got 99% full, and if you got it there you'd just get "disk is full" message instead of some catastrophe? I don't need that kind of stress in my life...

And with plain RAID6 I can still add hard drives if needed. Yeah the case of buying bigger drives is still a bit iffy but btrfs RAID1 gonna lose me more space than RAID6+LVM+xfs anyway...

Re: OpenZFS – add disks to existing RAIDZ

#167

Post by ilyt »

didntcheck wrote: post #125
ilyt wrote: post #72

Earlier quoted context omitted.

....vs stressing all of them for parity rebuild.

However RAIDZ2 can survive up to one of those stressed disks failing, while a 2-mirror has everything riding on that single survivor. It seems intuitive that mirrors would be safer, yeah, but if you run the binomial distribution numbers there are some realistic combinations of array size and individual drive failure probabilities where RAIDZ2 really does appear to be safer Imagine a gambler telling you "you can eithe…

Well, from my probe of 500 servers I have at work I have seen RAID1 rebuild failing zero times and RAID6 failing once (but we did recover it).

Granted, that was a bit of uncommon case where:

* someone forgot to order spares after taking last one

* we still had our consumable buying pipeline going thru helpdesk

* helpdesk didn't had any importance communicated about it and because of some accounting bullshit the purchase got delayed long enough

* the drives in question were all from some segate's fuckup of a model with much higher failure rates.

One disk failed with some media errors, remaining 2 got kicked out of array for same reason during resilvering

We ddrescue'd the 2 on the pair of fresh ones and the bad blocks didn't land in the same place on both drives so it made full recovery. But we did learn many lessons from that..

Re: OpenZFS – add disks to existing RAIDZ

#168

Post by ilyt »

edmundsauto wrote: post #149
ilyt wrote: post #144

Earlier quoted context omitted.

To replace drive in RAID5/6 array, you need to read the entirety of every drive To replace drive in RAID1 you need to read the entirety of one drive. Instead of reading one whole drive worth of stress, you're reading N drives worth of stress But it is funny that the people making arguments about "stressing drives" don't even fucking know how RAID works...

However, RAID1, you stress the drive that has your only copy of the data. With 5/6 (RAIDz w/ 2 spares as well) you can have multiple copies of the data, so although you are overall increasing stress on your fleet, wouldn't it have a lower probability of data loss? Drive stress (for me) is mostly a concern about data loss.

With RAID5 you'd have higher probability of data loss. Remember, you just loss the only redundancy you had, brining your redundancy to same level as RAID1 with failed drive.

... except now you have more drives that can fail.

With RAID6 yes, you can still fail one more time and still keep your data, which is why it is recommended.

Data safety wise I'd go RAID6 -> RAID1/10 (particularly linux implementation can do raid10 on odd number of drives which is nice) -> RAID5

Re: OpenZFS – add disks to existing RAIDZ

#169

Post by ilyt »

ErneX wrote: post #165
ilyt wrote: post #145

Earlier quoted context omitted.

That talks more about RAID5/Z1 vs mirroring. You still need 6/Z2 if you want to have reasonable fault tolerance, unless you want to waste ungodly amount of space.

My reasoning for using mirrored vdevs was more about the ease of expanding the storage than the redundancy.

We just used RAID6... expanding by more drives is easy while keeping data waste low and the "we want to put bigger drives now" case isn't all that problematic considering that what you save you waste on low efficiency of RAID1 setup.

We did had a big case so we just ran 2xRAID6 setup and that one time where it was needed we just replaced to bigger drives one by one, while using the removed ones as spares for other machines. But that's benefit of scale bigger than "a NAS server".

Re: OpenZFS – add disks to existing RAIDZ

#170

Post by yencabulator »

kalleboo wrote: post #84
jedberg wrote: post #30

I wish Apple and Oracle would have just sorted things out and made ZFS the main filesystem for the Mac. Way back in the day when they first designed Time Machine, it was supposed to just be a GUI for ZFS snapshots. How cool would it be if we had a great GUI for ZFS (snapshots, volume management, etc.). I could buy a new external disk, add it to a pool, have seamless storage expansion. It would be great. Ah, what coul…

> I could buy a new external disk, add it to a pool, have seamless storage expansion. You still can't do that with ZFS though, that's what this PR is for.

Beware using ZFS with external disks: https://github.com/openzfs/zfs/issues/3461
Post reply on HN