Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS 2.3 released with ZFS raidz expansion

github.com

Post reply on HN
Print view View original

141–150 of 331 posts

Re: ZFS 2.3 released with ZFS raidz expansion

#141

Post by crest »

azalemeth wrote: post #96
lousken wrote: post #74

Earlier quoted context omitted.

as far as stability goes, btrfs is used by meta, synology and many others, so I wouldn't say it's not stable, but some features are lacking

My understanding is that single-disk btrfs is good, but raid is decidedly dodgy; https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid5... states that: > The RAID56 feature provides striping and parity over several devices, same as the traditional RAID5/6. > There are some implementation and design deficiencies that make it unreliable for some corner cases and *the feature should not be used in production, only…

One problem with your setup is that ZFS by design can't use a traditional *nix filesystem buffer cache. Instead it has to use its own ARC (adaptive replacement cache) with end-to-end checksumming, transparent compression, and copy-on-write semantics. This can lead to annoying performance problems when the two types of file system caches contest for available memory. There is a back pressure mechanism, but it effectively pauses other writes while evicting dirty cache entries to release memory.

Re: ZFS 2.3 released with ZFS raidz expansion

#142

Post by pdimitar »

koverstreet wrote: post #94
mustache_kimono wrote: post #70

Earlier quoted context omitted.

> Technically speaking, bcachefs has been merged into the Linux Kernel - that makes your initial assertion wrong. bcachefs doesn't implement its erasure coding/RAID yet? Doesn't implement send/receive. Doesn't implement scrub/fsck. See: https://bcachefs.org/Roadmap , https://bcachefs.org/Wishlist/ btrfs is still more of a legit competitor to ZFS these days and it isn't close to touching ZFS where it matters. If the p…

Erasure coding is almost done; all that's missing is some of the device evacuate and reconstruct paths, and people have been testing it and giving positive feedback (especially w.r.t. performance). It most definitely does have fsck and has since the beginning, and it's a much more robust and dependable fsck than btrfs's. Scrub isn't quite done - I actually was going to have it ready for this upcoming merge window exc…

FYI: the main reason I gave up on bcachefs is that I can't use devices with native 16K blocks.

Hope that's coming this year. I have a bunch of old HDDs and SSDs and I could very easily assemble a spare storage server with about 4TB capacity. Already tested bcachefs with most of the drives and it performed very well.

Also lack of ability to reconstruct seems like another worrying omission.

Re: ZFS 2.3 released with ZFS raidz expansion

#143

Post by mafuy »

koverstreet wrote: post #94
mustache_kimono wrote: post #70

Earlier quoted context omitted.

> Technically speaking, bcachefs has been merged into the Linux Kernel - that makes your initial assertion wrong. bcachefs doesn't implement its erasure coding/RAID yet? Doesn't implement send/receive. Doesn't implement scrub/fsck. See: https://bcachefs.org/Roadmap , https://bcachefs.org/Wishlist/ btrfs is still more of a legit competitor to ZFS these days and it isn't close to touching ZFS where it matters. If the p…

Erasure coding is almost done; all that's missing is some of the device evacuate and reconstruct paths, and people have been testing it and giving positive feedback (especially w.r.t. performance). It most definitely does have fsck and has since the beginning, and it's a much more robust and dependable fsck than btrfs's. Scrub isn't quite done - I actually was going to have it ready for this upcoming merge window exc…

Thank you, looking forward to it!

Re: ZFS 2.3 released with ZFS raidz expansion

#144

Post by ryao »

Twey wrote: post #115
ekianjo wrote: post #81

Earlier quoted context omitted.

It provides encryption by default without having to deal with LUKS. And no need to ever do fsck again.

Except that swap on OpenZFS still deadlocks 7 years later ( https://github.com/openzfs/zfs/issues/7734 ) so you're still going to need LUKS for your swap anyway.

Another option is to go without swap. I avoid swap on my machines unless I want hibernation support.

Re: ZFS 2.3 released with ZFS raidz expansion

#145

Post by BSDobelix »

bayindirh wrote: post #127
poisonborz wrote: post #59

I just don't get it how the Windows world - by far the largest PC platform per userbase - still doesn't have any answer to ZFS. Microsoft had WinFS and then ReFS but it's on the backburner and while there is active development (Win11 ships some bits time to time) release is nowhere in sight. There are some lone warriors trying the giant task of creating a ZFS compatibility layer with some projects, but they are far f…

> How come that Windows still uses a 32 year old file system? Simple. Because most of the burden is taken by the (enterprise) storage hardware hosting the FS. Snapshots, block level deduplication, object storage technologies, RAID/Resiliency, size changes, you name it. Modern storage appliances are black magic, and you don't need much more features from NTFS. You either transparently access via NAS/SAN or store your…

>ZFS is mostly for resilient, but not performance critical needs.

Los Alamos disagrees ;)

https://www.lanl.gov/media/news/0321-computational-storage

But yes, in general you are right, Cern for example uses Ceph:

https://indico.cern.ch/event/1457076/attachments/2934445/515...

Re: ZFS 2.3 released with ZFS raidz expansion

#146

Post by crote »

chungy wrote: post #51
uniqueuid wrote: post #4

It's good to see that they were pretty conservative about the expansion. Not only is expansion completely transparent and resumable, it also maintains redundancy throughout the process. That said, there is one tiny caveat people should be aware of: > After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set o…

I'm not sure that's really a caveat, it just means old data might be in an inoptimal layout. Even with that, you still get the full benefits of raidzN, where up to N disks can completely fail and the pool will remain functional.

I think it's a huge caveat, because it makes upgrades a lot less efficient than you'd expect.

For example, home users generally don't want to buy all of their storage up front. They want to add additional disks as the array fills up. Being able to start with a 2-disk raidz1 and later upgrade that to a 3-disk and eventually 4-disk array is amazing. It's a lot less amazing if you end up with a 55% storage efficiency rather than 66% you'd ideally get from a 2-disk to 3-disk upgrade. That's 11% of your total disk capacity wasted, without any benefit whatsoever.

Re: ZFS 2.3 released with ZFS raidz expansion

#147

Post by ryao »

formerly_proven wrote: post #121
cm2187 wrote: post #13

Earlier quoted context omitted.

So for instance I have a ZFS pool with 3 HDD data vdevs, and 2 SSD special vdevs. I want to convert the two SSD vdevs into a single one (or possibly remove one of them). From what I read the only way to do that is to destroy the entire pool and recreate it (it's in a server in a datacentre, don't want to reupload that much data). In windows, you can set a disk for removal, and as long as the other disks have enough s…

My understanding is that ZFS does virtual physical translation in the vdev layer, i.e. all block references in ZFS contain a (vdev, vblock) tuple, and the vdev knows how to translate that virtual block offset into actual on-disk block offset(s). This kinda implies that you can't actually remove data vdevs, because in practice you can't rewrite all references. You also can't do offline deduplication without rewriting…

There is a clever trick that is used to make top level removal work. The code will make the vdev readonly. Then it will copy its contents into free space on other vdevs (essentially, the contents will be stored behind the scenes in a file). Finally, it will redirect reads on that vdev into the stored vdev. This indirection allows you to remove the vdev. It is not implemented for raid-z at present though.

Re: ZFS 2.3 released with ZFS raidz expansion

#148

Post by bayindirh »

BSDobelix wrote: post #145
bayindirh wrote: post #127

Earlier quoted context omitted.

> How come that Windows still uses a 32 year old file system? Simple. Because most of the burden is taken by the (enterprise) storage hardware hosting the FS. Snapshots, block level deduplication, object storage technologies, RAID/Resiliency, size changes, you name it. Modern storage appliances are black magic, and you don't need much more features from NTFS. You either transparently access via NAS/SAN or store your…

>ZFS is mostly for resilient, but not performance critical needs. Los Alamos disagrees ;) https://www.lanl.gov/media/news/0321-computational-storage But yes, in general you are right, Cern for example uses Ceph: https://indico.cern.ch/event/1457076/attachments/2934445/515...

I think what LLNL did predates GPUDirect and other new technologies came after 2022, but that's a good start.

CERN's Ceph also for their "General IT" needs. Their clusters are independent from that. Also CERN's most processing is distributed across Europe. We are part of that network.

Many, if not all of the HPC centers we talk with uses Lustre as their "immediate" storage. Also, there's Weka now, a closed source storage system supporting insane speeds and tons of protocols at the same time. Mostly used for and by GPU clusters around the world. You connect terabits to that cluster casually. It's all flash, and flat out fast.

Re: ZFS 2.3 released with ZFS raidz expansion

#149

Post by BSDobelix »

bhaney wrote: post #139
BSDobelix wrote: post #116

Earlier quoted context omitted.

>ZFS has license issues with Linux, preventing full integration No one wants that, openZFS is much healthier without Linux and it's "Foundation/Politics".

> No one wants that I want that

Then let me tell you that FreeBSD or OmniOS is what you really want ;)
Post reply on HN