Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS 2.3 released with ZFS raidz expansion

github.com

Post reply on HN
Print view View original

171–180 of 331 posts

Re: ZFS 2.3 released with ZFS raidz expansion

#171

Post by chillfox »

ryao wrote: post #62
endorphine wrote: post #25

Can someone describe why they would use ZFS (or similar) for home usage?

To give an answer that nobody else has given, ZFS is great for storing Steam games. Set recordsize=1M and compression=zstd and you can often store about 33% more games in the same space. A friend uses ZFS to store his Steam games on a couple of hard drives. He gave ZFS a SSD to use as L2ARC. ZFS automatically caches the games he likes to run on the SSD so that they load quickly. If he changes which games he likes to…

The compression and ARC will make games load much master than they would on NTFS even without having a separate drive for the ARC.

Re: ZFS 2.3 released with ZFS raidz expansion

#172

Post by bmicraft »

stavros wrote: post #82
Timshel wrote: post #80

Earlier quoted context omitted.

I don't believe it supports adding parity drives only data drives.

Ahh interesting, thanks.

Since preexisting blocks are kept at their current parity ratio and not modified (only redistributed among all devices), increasing the parity level of new blocks won't really be useful in practice anyway.

Re: ZFS 2.3 released with ZFS raidz expansion

#173

Post by bayindirh »

BSDobelix wrote: post #170
poisonborz wrote: post #161

Earlier quoted context omitted.

So private consumers should just pay cloud subscription if they want safer/modern data storage for their PC? (without NAS)

If you need Windows, you can use something like restic (checksums and compression) and external drives (more than one, stored in more than one place) to make a backup. Plus "maybe" but not needed ReFS (on your non-Windows partition), which is included in the Workstation/Enterprise editions of Windows. I trust my own backups much more than any subscription, not essentially from a technical point of view, but from an a…

> I trust my own backups much more than any subscription, not from a technical standpoint but from an access one (for example, losing access to your google account).

I personally use cloud storage extensively, but I keep a local version with periodic rclone/borg. It allows me access from everywhere and sleep well at night.

Re: ZFS 2.3 released with ZFS raidz expansion

#174

Post by qwertox »

BSDobelix wrote: post #170
poisonborz wrote: post #161

Earlier quoted context omitted.

So private consumers should just pay cloud subscription if they want safer/modern data storage for their PC? (without NAS)

If you need Windows, you can use something like restic (checksums and compression) and external drives (more than one, stored in more than one place) to make a backup. Plus "maybe" but not needed ReFS (on your non-Windows partition), which is included in the Workstation/Enterprise editions of Windows. I trust my own backups much more than any subscription, not essentially from a technical point of view, but from an a…

NTFS has Volume Shadow Copy, which is "good enough" for private users if they want to create image backups while their system is running.

Re: ZFS 2.3 released with ZFS raidz expansion

#175

Post by cesarb »

GuB-42 wrote: post #73
poisonborz wrote: post #59

I just don't get it how the Windows world - by far the largest PC platform per userbase - still doesn't have any answer to ZFS. Microsoft had WinFS and then ReFS but it's on the backburner and while there is active development (Win11 ships some bits time to time) release is nowhere in sight. There are some lone warriors trying the giant task of creating a ZFS compatibility layer with some projects, but they are far f…

To be honest, the situation with Linux is barely better. ZFS has license issues with Linux, preventing full integration, and Btrfs is 15 years in the making and still doesn't match ZFS in features and stability. Most Linux distros still use ext4 by default, which is 19 years old, but ext4 is little more than a series of extensions on top of ext2, which is the same age as NTFS. In all fairness, there are few OS compon…

> Btrfs [...] still doesn't match ZFS in features [...]

Isn't the feature in question (array expansion) precisely one which btrfs already had for a long time? Does ZFS have the opposite feature (shrinking the array), which AFAIK btrfs also already had for a long time?

(And there's one feature which is important to many, "being in the upstream Linux kernel", that ZFS most likely will never have.)

Re: ZFS 2.3 released with ZFS raidz expansion

#176

Post by BSDobelix »

qwertox wrote: post #174
BSDobelix wrote: post #170

Earlier quoted context omitted.

If you need Windows, you can use something like restic (checksums and compression) and external drives (more than one, stored in more than one place) to make a backup. Plus "maybe" but not needed ReFS (on your non-Windows partition), which is included in the Workstation/Enterprise editions of Windows. I trust my own backups much more than any subscription, not essentially from a technical point of view, but from an a…

NTFS has Volume Shadow Copy, which is "good enough" for private users if they want to create image backups while their system is running.

First of all, that's not a backup, that's a snapshot, and NO, that's not "good enough", tell your grandma that all her digitised pictures are gone because her hard drive exploded, or that one most important jpeg is now unwatchable because of bitrot.

Just because someone is a private user doesn't mean that the data is less important, often it's quite the opposite, for example a family album vs your cloned git repository.

Re: ZFS 2.3 released with ZFS raidz expansion

#177

Post by rabf »

ryao wrote: post #83
wil421 wrote: post #77

Earlier quoted context omitted.

How many of us are using single disks on our laptops? I have a NAS and use all of the above but that doesn’t help people with single drive systems. Or help me understand why I would want it on my laptop.

My thinkpad from college uses ZFS as its rootfs. The benefits are: * If the hard drive / SSD corrupted blocks, the corruption would be identified. * Ditto blocks allow for self healing. Usually, this only applies to metadata, but if you set copies=2, you can get this on data too. It is a poor man’s RAID. * ARC made the desktop environment very responsive since unlike the LRU cache, ARC resists cold cache effects from…

Not ever having to deal with partitions and instead using data sets each of which can have their own properties such as compression, size quota, encryption etc is another benefit. Also using zfsbootmenu instead of grub enables booting from different datasets or snapshots as well as mounting and fixing data sets all from the bootloader!

Re: ZFS 2.3 released with ZFS raidz expansion

#178

Post by shrubble »

poisonborz wrote: post #161
bayindirh wrote: post #127

Earlier quoted context omitted.

> How come that Windows still uses a 32 year old file system? Simple. Because most of the burden is taken by the (enterprise) storage hardware hosting the FS. Snapshots, block level deduplication, object storage technologies, RAID/Resiliency, size changes, you name it. Modern storage appliances are black magic, and you don't need much more features from NTFS. You either transparently access via NAS/SAN or store your…

So private consumers should just pay cloud subscription if they want safer/modern data storage for their PC? (without NAS)

No, private consumers have a choice, since Linux and FreeBSD runs well on their hardware. Microsoft is too busy shoveling their crappy AI and convincing OEMs to put a second Windows button (the CoPilot button) on their keyboards.

Re: ZFS 2.3 released with ZFS raidz expansion

#179

Post by ryao »

rekoil wrote: post #152
uniqueuid wrote: post #4

It's good to see that they were pretty conservative about the expansion. Not only is expansion completely transparent and resumable, it also maintains redundancy throughout the process. That said, there is one tiny caveat people should be aware of: > After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set o…

Yaeh it's a pretty huge caveat to be honest. Da1 Db1 Dc1 Pa1 Pb1 Da2 Db2 Dc2 Pa2 Pb2 Da3 Db3 Dc3 Pa3 Pb3 ___ ___ ___ Pa4 Pb4 ___ represents free space. After expansion by one disk you would logically expect something like: Da1 Db1 Dc1 Da2 Pa1 Pb1 Db2 Dc2 Da3 Db3 Pa2 Pb2 Dc3 ___ ___ ___ Pa3 Pb3 ___ ___ ___ ___ Pa4 Pb4 But as I understand it it would actually expand to: Da1 Db1 Dc1 Dd1 Pa1 Pb1 Da2 Db2 Dc2 Dd2 Pa2 Pb2 D…

ZFS RAID-Z does not have parity disks. The parity and data is interleaved to allow data reads to be done from all disks rather than just the data disks.

The slides here explain how it works:

https://openzfs.org/w/images/5/5e/RAIDZ_Expansion_2023.pdf

Anyway, you are not entirely wrong. The old data will have the old parity:data ratio while new data will have the new parity:data ratio. As old data is freed from the vdev, new writes will use the new parity:data ratio. You can speed this up by doing send/receive, or by deleting all snapshots and then rewriting the files in place. This has the caveat that reflinks will not survive the operation, such that if you used reflinks to deduplicate storage, you will find the deduplication effect is gone afterward.

Re: ZFS 2.3 released with ZFS raidz expansion

#180

Post by wkat4242 »

Note: This is online expansion. Expansion was always possible but you did need to take the array down to do it. You could also move to bigger drives but you also had to do that one at a time (and only gain the new capacity once all drives were upgraded of course)

As far as I know shrinking a pool is still not possible though. So if you have a pool with 5 drives and add a 6th, you can't go back to 5 drives even if there is very little data in it.

Post reply on HN