Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS 2.3 released with ZFS raidz expansion

github.com

Post reply on HN
Print view View original

231–240 of 331 posts

Re: ZFS 2.3 released with ZFS raidz expansion

#231

Post by ryao »

Dalewyn wrote: post #226
radlad wrote: post #225

Earlier quoted context omitted.

How can you achieve 2-disk fault tolerance using btrfs and RAID 1?

By using three drives. RAID1 is just making literal copies, so each additional drive in a RAID1 is a self-sufficient copy. You want two drives of fault tolerance? Use three drives, so if you lose two copies you still have one left. This is of course hideously inefficient as you scale larger, but that is not the question posed.

Btrfs did not support that until Linux 5.5 when it added RAID1c3. On its mirror devices instead of doing mirroring, it just stores 2 copies, no matter how many mirror members you have.

Re: ZFS 2.3 released with ZFS raidz expansion

#232

Post by ryao »

_joel wrote: post #136
lousken wrote: post #133

Earlier quoted context omitted.

I am assuming almost everybody at some point experienced data loss because they pulled out a flash drive too early. Is it safe to assume that we stopped using flash drives because of it?

I'm not sure we have stopped using flash, judging by the pile of USB sticks on my desk :) In relation to the fs analogy if you used a flash drive that you know corrupted your data, you'd throw it away for one you know works.

I once purchased a bunch of flash drives from Google’s online swag store and just unplugging them was often enough to put then in a state where they claimed to be 8MB devices and nothing I wrote to them was ever possible to read back in my limited tests. I stopped using those fast.

Re: ZFS 2.3 released with ZFS raidz expansion

#233

Post by radlad »

Dalewyn wrote: post #226
radlad wrote: post #225

Earlier quoted context omitted.

How can you achieve 2-disk fault tolerance using btrfs and RAID 1?

By using three drives. RAID1 is just making literal copies, so each additional drive in a RAID1 is a self-sufficient copy. You want two drives of fault tolerance? Use three drives, so if you lose two copies you still have one left. This is of course hideously inefficient as you scale larger, but that is not the question posed.

> This is of course hideously inefficient as you scale larger, but that is not the question posed.

It's not just inefficient, you literally can't scale larger. Mirroring is all that RAID 1 allows for. To scale, you'd have to switch to RAID 10, which doesn't allow two-disk fault tolerance (you can get lucky if they are in different stripes, but this isn't fault tolerance.)

But you're right - RAID 1 also scales terribly compared to RAID 6, even before introducing striping. Imagine you have 6 x 16 TB disks:

With RAID 6, usable space of 64 TB, two-drive fault tolerance.

With RAID 1, usable space of 16 TB, five-drive fault tolerance.

With RAID 10, usable space of 32 GB, one-drive fault tolerance.

Re: ZFS 2.3 released with ZFS raidz expansion

#234

Post by ryao »

bayindirh wrote: post #129
GuB-42 wrote: post #73

Earlier quoted context omitted.

To be honest, the situation with Linux is barely better. ZFS has license issues with Linux, preventing full integration, and Btrfs is 15 years in the making and still doesn't match ZFS in features and stability. Most Linux distros still use ext4 by default, which is 19 years old, but ext4 is little more than a series of extensions on top of ext2, which is the same age as NTFS. In all fairness, there are few OS compon…

> Most Linux distros still use ext4 by default, which is 19 years old, but ext4 is little more than a series of extensions on top of ext2, which is the same age as NTFS. However, ext4 and XFS are much more simpler and performant than BTRFS & ZFS as root drives on personal systems and small servers. I personally won't use either on a single disk system as root FS, regardless of how fast my storage subsystem is.

ZFS will outscale ext4 in parallel workloads with ease. XFS will often scale better than ext4, but if you use L2ARC and SLOG devices, it is no contest. On top of that, you can use compression for an additional boost.

You might also find ZFS outperforms both of them in read workloads on single disks where ARC minimizes cold cache effects. When I began using ZFS for my rootfs, I noticed my desktop environment became more responsive and I attributed that to ARC.

Re: ZFS 2.3 released with ZFS raidz expansion

#235

Post by ryao »

wkat4242 wrote: post #214
unsnap_biceps wrote: post #210

Earlier quoted context omitted.

Raidz wasn't able to be expanded in place before this. You were able to add to a pool that included a raidz vdev, but that raidz vdev was immutable.

Oh ok, I've never done this, but I thought it was already there. Maybe this was the original ZFS from Sun? But maybe I just remember it incorrectly, sorry. I've used it on multi-drive arrays but I never had the need for expansion.

You could add top level raidz vdevs or replace the members of a raid-z vdev with larger disks to increase storage space back then. You still have those options now.

Re: ZFS 2.3 released with ZFS raidz expansion

#236

Post by bayindirh »

ryao wrote: post #234
bayindirh wrote: post #129

Earlier quoted context omitted.

> Most Linux distros still use ext4 by default, which is 19 years old, but ext4 is little more than a series of extensions on top of ext2, which is the same age as NTFS. However, ext4 and XFS are much more simpler and performant than BTRFS & ZFS as root drives on personal systems and small servers. I personally won't use either on a single disk system as root FS, regardless of how fast my storage subsystem is.

ZFS will outscale ext4 in parallel workloads with ease. XFS will often scale better than ext4, but if you use L2ARC and SLOG devices, it is no contest. On top of that, you can use compression for an additional boost. You might also find ZFS outperforms both of them in read workloads on single disks where ARC minimizes cold cache effects. When I began using ZFS for my rootfs, I noticed my desktop environment became mo…

No doubt. I want to reiterate my point. Citing myself:

> "I personally won't use either on a single disk system as root FS, regardless of how fast my storage subsystem is." (emphasis mine)

We are no strangers to filesystems. I personally benchmarked a ZFS7320 extensively, writing a characterization report, plus we have a ZFS7420 for a very long time, complete with separate log SSDs for read and write on every box.

However, ZFS is not saturation proof, plus is nowhere near a Lustre cluster performance wise, when scaled.

What kills ZFS and BTRFS on desktop systems are write performance, esp. on heavy workloads like system updates. If I need a desktop server (performance-wise), I'd configure it accordingly and use these, but I'd never use BTRFS or ZFS on a single root disk due to their overhead, to reiterate myself thrice.

Re: ZFS 2.3 released with ZFS raidz expansion

#237

Post by parshimers »

shepherdjerred wrote: post #205

How does ZFS compare to btrfs? I'm currently using btrfs for my home server, but I've had some strange troubles with it. I'm thinking about switching to ZFS, but I don't want to end up in the same situation.

btrfs has similar aims to ZFS, but is far less mature. i used it for my root partitions due to it not needing DKMS, but had many troubles. i used it in a fairly simple way, just a mirror. one day, of the drives in the array started to have issues- and btrfs fell on it's face. it remounted everything read-only if i remember correctly, and would not run in degraded mode by default. even mdraid would do better than this without checksumming and so forth. ZFS also likewise, says that the array is faulted, but of course allows it to be used. the fact the default behavior was not RAID, because it's literally missing the R part for reading the data back, made me lose any faith in it. i moved to ZFS and haven't had issues since. there is much more of a community and lots of good tooling around it.

Re: ZFS 2.3 released with ZFS raidz expansion

#238

Post by bayindirh »

ryao wrote: post #228
bayindirh wrote: post #148

Earlier quoted context omitted.

I think what LLNL did predates GPUDirect and other new technologies came after 2022, but that's a good start. CERN's Ceph also for their "General IT" needs. Their clusters are independent from that. Also CERN's most processing is distributed across Europe. We are part of that network. Many, if not all of the HPC centers we talk with uses Lustre as their "immediate" storage. Also, there's Weka now, a closed source sto…

Did you confuse LANL for LLNL?

It's just a typo, not a confusion, and I'm well beyond the edit window.

Re: ZFS 2.3 released with ZFS raidz expansion

#239

Post by ryao »

pdimitar wrote: post #216
koverstreet wrote: post #201

Earlier quoted context omitted.

I wasn't aware there were actual users needing bs > ps yet. Cool :) That should be a completely trivial for bcachefs to support, it'll mostly just be a matter of finding or writing the tests.

Seriously? But... NVMe drives! I stopped testing because I only have one spare NVMe and couldn't use it with bcachefs. If you or others can get it done I'm absolutely starting to use bcachefs the month after. I do need fast storage servers in my home office.

You can do this on ZFS today with `zpool create -o ashift=14 ...`.

Re: ZFS 2.3 released with ZFS raidz expansion

#240

Post by pdimitar »

ryao wrote: post #239
pdimitar wrote: post #216

Earlier quoted context omitted.

Seriously? But... NVMe drives! I stopped testing because I only have one spare NVMe and couldn't use it with bcachefs. If you or others can get it done I'm absolutely starting to use bcachefs the month after. I do need fast storage servers in my home office.

You can do this on ZFS today with `zpool create -o ashift=14 ...`.

Yeah I know, thanks. But ZFS still mostly requires drives with the same sizes. My main NAS is like that but I can't expand it even though I want to, with drives of different sizes I have lying around, and I am not keen on spending for new HDDs right now. So I thought I'll make a secondary NAS with bcachefs and all the spare drives I have.

As for ZFS, I'll be buying some extra drives later this year and will make use of direct_io so I can use another NVMe spare for faster access.

Post reply on HN