Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS – add disks to existing RAIDZ

github.com

Post reply on HN
Print view View original

131–140 of 177 posts

Re: OpenZFS – add disks to existing RAIDZ

#131

Post by someplaceguy »

didntcheck wrote: post #129
keeperofdakeys wrote: post #99

Earlier quoted context omitted.

A mirror resilver is a relatively linear and sequential rewrite, so its very fast. Raidz resilvers require lots of random reads and writes across all the drives, and requires waiting for all drives to read data before it can be written to the replaced drive - "herding cats" sounds appropriate here.

That makes sense. Usually the received wisdom I hear is "RAIDZ is slower due to the parity calculations", which has always seemed dubious to me on modern CPUs, given we can do much more complex compression and encryption at way faster then disk speed (for HDDs, at least)

Usually what makes RAID-Z slower is not really the parity calculations, it's the fact that each RAID-Z vdev only has the read IOPS of the slowest disk in it.

So for example, a pool with 2 RAID-Z vdevs each containing 5 disks effectively has only 2x the IOPS of a single disk, while a pool with 5 mirror vdevs (i.e. RAID-10) has 10x the IOPS of a single disk.

It's not a big problem if you mostly do sequential I/O but it's a huge difference if/when you do small random reads (e.g. traversing a non-cached directory tree).

In the context of resilvers, currently RAID-Z pools can only be done by traversing the block tree which, due to fragmentation of CoW filesystems, usually leads to a lot of random reads/writes, while a RAID-10 pool can basically resilver a pool while doing almost fully sequential I/O which can be much, much faster.

Re: OpenZFS – add disks to existing RAIDZ

#132

Post by wkat4242 »

mekster wrote: post #112
wkat4242 wrote: post #83

Earlier quoted context omitted.

Yeah I wonder if it was really caused by Jonathan Schwartz bragging to the media about it and pissing Steve Jobs off, as was rumoured at the time. Or if there was more behind it. I would certainly not put it past Jobs. While he had some genius qualities, his personality was pretty deeply flawed. But on the other hand I wonder if his sense for business would not have prevailed. If only Apple had absorbed sun instead o…

Not sure why you limit that to desktop. It's more important that server files can have an instant snapshot, compression and performant incremental backup to remote servers.

Because the discussion was about Mac and macOS doesn't have a server version anymore that's worth speaking about.

But when all this happened they still did, yes.

Re: OpenZFS – add disks to existing RAIDZ

#133

Post by toyg »

wkat4242 wrote: post #83
jedberg wrote: post #30

I wish Apple and Oracle would have just sorted things out and made ZFS the main filesystem for the Mac. Way back in the day when they first designed Time Machine, it was supposed to just be a GUI for ZFS snapshots. How cool would it be if we had a great GUI for ZFS (snapshots, volume management, etc.). I could buy a new external disk, add it to a pool, have seamless storage expansion. It would be great. Ah, what coul…

Yeah I wonder if it was really caused by Jonathan Schwartz bragging to the media about it and pissing Steve Jobs off, as was rumoured at the time. Or if there was more behind it. I would certainly not put it past Jobs. While he had some genius qualities, his personality was pretty deeply flawed. But on the other hand I wonder if his sense for business would not have prevailed. If only Apple had absorbed sun instead o…

Apple buying Sun would have been a disaster. Oracle at least needed some (arguably most) of what Sun made - Apple did not and do not.

Apple would have killed Java in half a second. They don't need it, it was built to do the opposite of what they want their tech to do. Same for Solaris, burdensome duplicate. As for all the desktop-oriented FOSS projects (OpenOffice, VirtualBox etc), they would not have just been killed - Apple would have viciously bullied anyone to preclude them from forking them. The stuff they'd keep (MySql, probably), they would have made osx-only, and then strangled them quietly after they got out of the server game. They would have ripped anything technologically interesting from the hardware divisions and then shut them down - because they were no match, in terms of supply efficiency and value, for the Apple equivalents; and anyway they were never seriously interested in the server space.

Oracle did what Oracle does, and realistically the culture clash was never going to result in a smooth transition; but from a commercial perspective they kept around the best of what Sun was making (Java, servers) and largely let the community get on with the stuff they weren't interested in (if rebranded/forked). The only real crime they committed, IMHO, was trying too hard to make money from VirtualBox, effectively spooking the market; there is a different timeline where VBox ends up being what Docker became. But everything else was par for the course.

Re: OpenZFS – add disks to existing RAIDZ

#134

Post by panick21_ »

jedberg wrote: post #30

I wish Apple and Oracle would have just sorted things out and made ZFS the main filesystem for the Mac. Way back in the day when they first designed Time Machine, it was supposed to just be a GUI for ZFS snapshots. How cool would it be if we had a great GUI for ZFS (snapshots, volume management, etc.). I could buy a new external disk, add it to a pool, have seamless storage expansion. It would be great. Ah, what coul…

ZFS was open source, Nothing was preventing Mac using ZFS. Just as they use DTrace. Its just Apple NIH.

Re: OpenZFS – add disks to existing RAIDZ

#135

Post by mustache_kimono »

toast0 wrote: post #81
mustache_kimono wrote: post #42

Earlier quoted context omitted.

> Does anyone know why this is the case? > Considering the process is already reading and rewriting basically the entire array, why not recalculate the parity as well? IANA expert but my guess is -- because, here, you don't have to modify block pointers, etc. ZFS RAIDZ is not like traditional RAID, as it's not just a sequence of arbitrary bits, data plus parity. RAIDZ stripe width is variable/dynamic, written in bloc…

> To do what you want, you can do what one could always do -- zfs send/recv between a filesystem between and old and new filesystem. Sure, but that involves having enough spare disks, enough places to put them, and enough places to connect them. This way, while the initial expansion is not ideal, it works. If you really need the space gains from a wider distribution, you can do this expansion and then do the make a n…

> Sure, but that involves having enough spare disks

So long as we are pointing out things: this presumes that the user doesn't have enough space to make the copies on their own disks? If one has a 3GB dataset and 9GB of free space, one can easily zfs send/recv that dataset and destroy the old copy.

> This way, while the initial expansion is not ideal, it works.

Yep.

> If you really need the space gains from a wider distribution, you can do this expansion and then do the make a new copy, then replace old copy with a new copy dance... although that's counter productive if you have snapshots.

That's why you do a zfs send/recv, instead of a copy. ZFS will copy your snapshots for you!

Re: OpenZFS – add disks to existing RAIDZ

#136

Post by mustache_kimono »

SpecialistK wrote: post #115
mustache_kimono wrote: post #51

Earlier quoted context omitted.

> How cool would it be if we had a great GUI for ZFS (snapshots, volume management, etc.). How cool would it be if we had a great TUI for ZFS... Live in the now: https://github.com/kimono-koans/httm

Now that's a smooth segue to self-promo. I respect it.

httm has been at 9XX GitHub stars for what seemed like forever! Time to get over that 1k hump.

As a Rust project, it's something of a duty to hype and self-promote, and to troll C++, of course. So let it be known: If you're not using httm, you're not using modern ZFS. Or -- httm is the cppfront of ZFS utilities? Or -- httm is the 'auto' keyword for filesystems, but less confusing (to me)?

I'll work on it.

Re: OpenZFS – add disks to existing RAIDZ

#137

Post by thrtythreeforty »

Filligree wrote: post #13
gigatexal wrote: post #10

Earlier quoted context omitted.

I’m holding out for bcachefs and still building ZFS via dkms on current kernels like a madman.

I’m running bcachefs on my desktop right now. It’s promising, but there’s… bugs. Right now only performance oriented ones, that I’ve noticed, but I’d wait a bit longer.

Can you elaborate on what kind of bugs? I've been following bcachefs casually because I am very excited for its eventual stabilization. But I am very conservative with my main storage array.

Re: OpenZFS – add disks to existing RAIDZ

#138

Post by rleigh »

katbyte wrote: post #69
crote wrote: post #29

> After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks. New blocks will be written with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been expanded once to 6-wide, has 4 data to 2 parity). Does anyone know why this is the case? When expanding an array which is getting full this wi…

I imagine there would eventually be a way / option to automatically rewrite old blocks. Or at least I would hope so because usually when your adding new disks the array is going to be near full

You can do a zfs send/zfs recv to send a dataset (including all snapshots) to yourself which is effectively rewriting the whole dataset, including all of its history, by duplicating it.

Not hard, but it does require sufficient free space. Once it's done you can destroy the original dataset and reclaim the space.

Re: OpenZFS – add disks to existing RAIDZ

#139

Post by terinjokes »

thrtythreeforty wrote: post #137
Filligree wrote: post #13

Earlier quoted context omitted.

I’m running bcachefs on my desktop right now. It’s promising, but there’s… bugs. Right now only performance oriented ones, that I’ve noticed, but I’d wait a bit longer.

Can you elaborate on what kind of bugs? I've been following bcachefs casually because I am very excited for its eventual stabilization. But I am very conservative with my main storage array.

Not GP, but I tried it out a few weeks ago. I experienced it working for about 15-20 minutes, then it stopped doing any I/O to the disks until I rebooted. After it happened a few times in one evening, I switched to something else.
Post reply on HN