Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS 2.3 released with ZFS raidz expansion

github.com

Post reply on HN
Print view View original

281–290 of 331 posts

Re: ZFS 2.3 released with ZFS raidz expansion

#281

Post by ryao »

BodyCulture wrote: post #248
scrp wrote: post #2

After years in the making ZFS raidz expansaion is finally here. Major features added in release: - RAIDZ Expansion: Add new devices to an existing RAIDZ pool, increasing storage capacity without downtime. - Fast Dedup: A major performance upgrade to the original OpenZFS deduplication functionality. - Direct IO: Allows bypassing the ARC for reads/writes, improving performance in scenarios like NVMe devices where cachi…

How well tested is this in combination with encryption? Is the ZFS team handling encryption as a first class priority at all? ZFS on Linux inherited a lot of fame from ZFS on Solaris, but everyone using it in production should study the issue tracker very well for a realistic impression of the situation.

The new features should interact fine with encryption. They are implemented at different parts of ZFS' internal stack.

There have been many man hours put into investigating bug reports involving encryption and some fixes were made. Unfortunately, something appears to be going wrong when non-raw sends of encrypted datasets are received by another system:

https://github.com/openzfs/zfs/issues/12014

I do not believe anyone has figured out what is going wrong there. It has not been for lack of trying. Raw sends from encrypted datasets appear to be fine.

Re: ZFS 2.3 released with ZFS raidz expansion

#282

Post by ryao »

p_l wrote: post #250
BodyCulture wrote: post #248

Earlier quoted context omitted.

How well tested is this in combination with encryption? Is the ZFS team handling encryption as a first class priority at all? ZFS on Linux inherited a lot of fame from ZFS on Solaris, but everyone using it in production should study the issue tracker very well for a realistic impression of the situation.

Main issue with encryption is occasional attempts by certain (specific) Linux kernel developer to lockout ZFS out of access to advanced instruction set extensions (far from the only weird idea of that specific developer). The way ZFS encryption is layered, the features should be pretty much orthogonal from each other, but I'll admit that there's a bit of lacking with ZFS native encryption (though mainly in upper laye…

These are actually wrappers around CPU instructions, so what ZFS does is implement its own equivalents. This does not affect encryption (beyond the inconvenience that we did not have SIMD acceleration for a while on certain architectures).

Re: ZFS 2.3 released with ZFS raidz expansion

#283

Post by artificialLimbs »

rabf wrote: post #177
ryao wrote: post #83

Earlier quoted context omitted.

My thinkpad from college uses ZFS as its rootfs. The benefits are: * If the hard drive / SSD corrupted blocks, the corruption would be identified. * Ditto blocks allow for self healing. Usually, this only applies to metadata, but if you set copies=2, you can get this on data too. It is a poor man’s RAID. * ARC made the desktop environment very responsive since unlike the LRU cache, ARC resists cold cache effects from…

Not ever having to deal with partitions and instead using data sets each of which can have their own properties such as compression, size quota, encryption etc is another benefit. Also using zfsbootmenu instead of grub enables booting from different datasets or snapshots as well as mounting and fixing data sets all from the bootloader!

Alright that's a bit mind blowing. TIL. Thank you. =)

Re: ZFS 2.3 released with ZFS raidz expansion

#284

Post by justinclift »

ryao wrote: post #32
lutorm wrote: post #29

Earlier quoted context omitted.

Although "Top-level vdevs can only be removed if the primary pool storage does not contain a top-level raidz vdev, all top-level vdevs have the same sector size, and the keys for all encrypted datasets are loaded."

I forgot we still did not have that last bit implemented. However, it is less important now that we have expansion.

> However, it is less important now that we have expansion.

Not really sure if that's true. They seem like two different/distinct use cases, though there's probably some small overlap.

Re: ZFS 2.3 released with ZFS raidz expansion

#285

Post by justinclift »

ryao wrote: post #55
Sesse__ wrote: post #54

Earlier quoted context omitted.

> I am not sure if reshaping is a reasonable thing. Yet people are celebrating when ZFS adds it. Was it all for nothing?

People wanted it, but it was very hard to do safely. While ZFS now can do it safely, many other storage solutions cannot. Those corruption issues I mentioned, where the RAID controller has no idea what to do, affect far more than just reshaping. They affect traditional RAID arrays when disks die and when patrol scrubs are done. I have not tested MD RAID on edge cases lately, but the last time I did, I found MD RAID i…

> While ZFS now can do it safely ...

It's the first release with the code, so "safely" might not be the right description until a few point releases happen. ;)

Re: ZFS 2.3 released with ZFS raidz expansion

#286

Post by JZerf »

craftkiller wrote: post #109
dizhn wrote: post #78

Earlier quoted context omitted.

Requirement for enterprise quality disks, huge RAM (1 gig per TB), ECC, at least x5 disks of redundancy. None of these are things, but people will try to educate you anyway. So use it but keep it to yourself. :)

No need to keep it to yourself. As you've mentioned, all of these requirements are misinformation so you can ignore people who repeat them (or even better, tell them to stop spreading misinformation). For those not in the know: You don't need to use enterprise quality disks. There is nothing in the ZFS design that requires enterprise quality disks any more than any other file system. In fact, ZFS has saved my data th…

One reason why it might be a good idea to use higher quality drives when using ZFS is because it seems like in some scenarios ZFS can result in more writes being done to the drive than when other file systems are used. This can be a problem for some QLC and TLC drives that have low endurance.

I'm in the process of setting up a server at home and was testing a few different file systems. I was doing a test where I had a program continuously synchronously writing just a single byte every second (like might happen for some programs that are writing logs fairly continuously). For most of my tests I was just using the default settings for each file system. When using ext4 this resulted in 28 KB/s of actual writes being done to the drive which seems reasonable due to 4 KB blocks needing to be written, journaling, writing metadata, etc... BTRFS generated 68 KB/s of actual writes which still isn't too bad. When using ZFS about the best I could get it to do after trying various settings for volblocksize, ashift, logbias, atime, and compression settings still resulted in 312 KB/s of actual writes being done to the drive which I was not pleased with. At the rate ZFS was writing data, over a 10 year span that same program running continuously would result in about 100 TB of writes being done to the drive which is about a quarter of what my SSD is rated for.

Re: ZFS 2.3 released with ZFS raidz expansion

#287

Post by unixhero »

zelcon wrote: post #274
unixhero wrote: post #45

Earlier quoted context omitted.

Just have backups. I used btrfs and zfs for different purposes. Never had any lost data or downtime with btrfs since 2016. I only use raid 0 and raid 1 and compression. Btrfs does not havr a hungry ram requirement.

Tbh the idea of keeping backups defeats the purpose of using RAIDZ (especially RAIDZ3). I don’t want to buy an LTO drive, so if I backup, it’s either buying more HDDs or S3 Glacier ($$$). I like RAIDZ so I don’t have to buy so many drives. I guess it protects you if your house burns down, but how many people do offsite backups for their personal files? And dormant, unpowered HDDs die a lot faster than live, powered H…

Yes, seriously handling your data is expensive. I am talking about buying new hardrives.

Re: ZFS 2.3 released with ZFS raidz expansion

#288

Post by menaerus »

ryao wrote: post #253
jeltz wrote: post #244

Earlier quoted context omitted.

Not on most database workloads. There zfs does not scale very well.

Percona and many others who benchmarked this properly would disagree with you. Percona found that ext4 and ZFS performed similarly when given identical hardware (with proper tuning of ZFS): https://www.percona.com/blog/mysql-zfs-performance-update/ In this older comparison where they did not initially tune ZFS properly for the database, they found XFS to perform better, only for ZFS to outperform it when tuning was d…

Refuting the "it doesn't scale" argument with a data from a blog that showcases a single workload (TPC-C) with 200G+10tables dataset (small to medium) at 2vCPU (wtf) machine with 16 connections (no thread pool so overprovisioned) is not quite a definition of a scale at all. It's a lost experiment if anything.

Re: ZFS 2.3 released with ZFS raidz expansion

#289

Post by bayindirh »

ryao wrote: post #258
bayindirh wrote: post #236

Earlier quoted context omitted.

No doubt. I want to reiterate my point. Citing myself: > "I personally won't use either on a single disk system as root FS , regardless of how fast my storage subsystem is." (emphasis mine) We are no strangers to filesystems. I personally benchmarked a ZFS7320 extensively, writing a characterization report, plus we have a ZFS7420 for a very long time, complete with separate log SSDs for read and write on every box. H…

I am generally happy with the write performance of ZFS. I have not noticed slow system updates on ZFS (although I run Gentoo, so slow is relative here). In what ways is the write performance bad? I am one of the OpenZFS contributors (although I am less active as late). If you bring some deficiency to my attention, there is a chance I might spend the time needed to improve upon it. By the way, ZFS limits the outstandi…

What I see with CoW filesystems is, when you force the FS to sync a lot (like apt does to keep immunity against power losses to a maximum), the write performance slouches visibly. This also means that when you're writing a lot of small files with a lot of processes and flood the FS with syncs, you get the same slouching, making everything slower in the process. This effect is better controlled in simpler filesystems, namely XFS and EXT4. This is why I keep backups elsewhere and keep my single disk rootfs on "simple" filesystems.

I'll be installing a 2 disk OpenZFS RAID1 volume on a SBC for high value files soon-ish, and I might be doing some tests on that when it's up. Honestly, I don't expect stellar performance since I'll be already putting it on constrained hardware, but let you know if I experience anything that doesn't feel right.

Thanks for the doc links, I'll be devouring them when my volume is up and running.

Where do you prefer your (bug and other) reports? GitHub? E-mail? IP over Avian Carriers?

Re: ZFS 2.3 released with ZFS raidz expansion

#290

Post by gf000 »

pimeys wrote: post #114
whataguy wrote: post #113

Earlier quoted context omitted.

The difference is that the ZFS kernel module is included by default with Proxmox, whereas with e.g. Debian, you would need to install it manually.

And you can't follow the latest kernel before the ZFS module supports it.

I use NixOS, and it simply updates to the latest kernel that supports zfs, with a single, declerative option.
Post reply on HN