Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of ZFS on Linux

clusterhq.com

Post reply on HN
Print view View original

111–120 of 125 posts

Re: The State of ZFS on Linux

#111

Post by ansible »

ryao wrote: post #106
colin_mccabe wrote: post #101

Earlier quoted context omitted.

As others have commented, CRC32 or CRC64 being "weak" in a cryptographic sense doesn't mean it's not suitable for detecting disk errors. x86 has a crc32 instruction, making it far faster than any other option. By the way, TCP doesn't use CRC32... it uses a simple additive checksum. Chris Mason wrote most of btrfs while he was at Oracle. Hence, the GPLv2 grants an implicit patent license to whatever patents Oracle mig…

Saying that a checksum is weak in a cryptographic sense would mean that it is possible for a malicious intelligence to generate collisions. This is not what I meant when I said that CRC was weak. Instead, I meant that it is trivial for ordinary glitches to corrupt data in ways that the checksums fail to detect. Things byte swaps and non-adjacent double bit flips are all that is necessary. The link I provided elaborat…

This is not what I meant when I said that CRC was weak. Instead, I meant that it is trivial for ordinary glitches to corrupt data in ways that the checksums fail to detect. Things byte swaps and non-adjacent double bit flips are all that is necessary. The link I provided elaborates on this ...

Since we're talking about data corruption that has managed to not be detected by the hard drive's own firmware (a read error not detected as a read error), I now wonder what sorts of data errors might also slip by the 32-bit CRC used by btrfs.

It would be hilarious (in a bad way) if the same algorithm was used at both levels. So that the same byte swap (for example) would exactly bypass both the drive firmware and the check done by btrfs.

If the algorithms are different though, then I'd expect the chances are reduced that a particular error can slip by both.

Re: The State of ZFS on Linux

#112

Post by mbreese »

DiabloD3 wrote: post #45
justincormack wrote: post #42

Earlier quoted context omitted.

There is bhyve now...

I've heard mixed reviews of that. It is probably not quite ready for production, but it will be interesting in the future.

Just to add to the reviews - I've had difficulty in getting Bhyve to consistently work. Sometimes the VM will boot fine, others it will just crash halfway into the boot cycle. When it works, it's good. It has a few years before it gets to Xen/KVM levels of stability, but it's a really good start.

Re: The State of ZFS on Linux

#113

Post by cmurf »

ryao wrote: post #107
cmurf wrote: post #98

Earlier quoted context omitted.

Metadata/data checksums are CRC-32C on all platforms and is per 4KB fs block. And ext4's (optional) checksumming also uses it. TCP is 16-bit and not CRC, although ethernet makes use of CRC-32. While SHA-2 is a cryptographic function and CRC-32C is not, and therefore inferior, as a checksum in the context of mostly but not entirely trusted hardware is adequate. It's also fast to the degree on modern hardware there's n…

You are correct. The ethernet frames use CRC32 while the IP frames use 16-bit CRC. While the checksums need not be of cryptographic quality (and not all checksum algorithms in ZFS are), there are multiple trivial ways in which the wrong data can match a CRC32 checksum and that is inadequate for the purposes of data integrity. It does help, but it does not go far enough. As for my assertion that btrfs uses 64-bit CRC…

I think the triviality and inadequacy are overstatements for the following two reasons: it's rare to have undetected errors in normally operating hardware that simultaneously go undetected by Btrfs. If this doesn't go far enough, then I'm challenged how to characterize the vast majority of the world's data on NTFS, HFS+, ext234, and XFS as being anything but fragile and tenuous, which would be an overstatement. What is going far enough? It's a use case question rather than something absolutely determinable. For a general purpose filesystem, I think CRC-32C is adequate, although it will be nice when there's choice for another algorithm to meet other use case requirements.

Yes, it won't send writable snapshots. e.g. ERROR: /mnt/root is not read-only.

Re: The State of ZFS on Linux

#114

Post by ClashTheBunny »

voltagex_ wrote: post #89
ryao wrote: post #52

Earlier quoted context omitted.

Below are a few remarks: To expand on your LZ4 comment, LZ4 is 3 times faster than LZO at decompression: https://code.google.com/p/lz4/ However, LZ4's real innovation is on incompressible data. I do not have a reference at the moment, but there have been benchmarks on the userland lz4 tool on incompressible data that show it processing 10GB/sec. This is because LZ4 uses a hash table that enables it to give up when tr…

Is it possible to integrate the ZoL modules with DKMS so that kernel upgrades are handled automagically?

That's exactly how it is set up. Just install a couple deb files or set up the repo and DKMS does the rest on every distro I've tried it on.

Re: The State of ZFS on Linux

#115

Post by mrmondo »

thijsb wrote: post #77
mrmondo wrote: post #73

While I like most parts of ZFS, these days BTRFS is both stable and performs well with a decent feature set. We moved from ZFS and EXT4 to BTRFS for a good portion of our production servers last year - and we haven't looked back.

Do you run RAID5/6? I had that running for half year, and it crashed often. Now on ZFS (raidz) and it works flawless

We run it on iSCSI LUNs straight from our SAN so this hasn't been an issue for us.

Re: The State of ZFS on Linux

#116

Post by rodgerd »

click170 wrote: post #85
rodgerd wrote: post #64

Earlier quoted context omitted.

> (which was Oracle's ZFS competitor that seems to be largely abandoned although I see commits in the kernel changelog periodically) Under heavy development, officially supported by most of the major commercial distros, and still designated by Linus as the ext* replacement as the standard Linux filesystem.

I'm curious where the idea that it's largely abandoned comes from. This isn't the first time I've heard it, but each time I've heard it and looked, the project looks far from dead or abandoned.

It seems to be a talking point amongst folks who want everyone to ignore Sun (and subsequently Oracle's) determination to keep ZFS out of Linux. If there's a healthy btrfs project then the constant stream of suggestions Linux distros should just ignore the huge legal problems associated with bundling ZFS seems like a completely insane idea.

Re: The State of ZFS on Linux

#117

Post by ryao »

Sanddancer wrote: post #46
ryao wrote: post #2

I am the author. Feel free to respond with questions. I will be watching for questions throughout the day.

What's performance like on ZoL compared to Solaris/OpenIndiana/FreeBSD?

Performance is a difficult question to answer because it is very workload dependent. However, I can say a few things on performance:

1. Solaris's EULA prohibits the publication of benchmarks without the explicit permission of Oracle. I have not done any benchmarks of Solaris and if I had done any benchmarks, I could not publish them without placing myself and/or my employer at risk of a lawsuit from Oracle.

2. Block device drivers are known to influence performance. In situations where Linux has superior driver support, ZoL will outperform its counterparts on other platforms.

3. There is a key performance fix in a pull request that is planned to be included in 0.6.4. I have seen benchmarks where it enables ZoL to outperform its counterparts on other platforms, although the other platforms were subject to inferior block device drivers. Before the fix, ZoL performed worse:

https://github.com/zfsonlinux/spl/pull/369

4. I have posted some information on performance to the ClusterHQ blog:

https://clusterhq.com/blog/state-zfs-on-linux/#comment-15844... https://clusterhq.com/blog/state-zfs-on-linux/#comment-15880...

5. A list of performance improvements in Open ZFS is available from the Open ZFS wiki:

http://open-zfs.org/wiki/Features#Performance

6. It is quite likely that I will post a future blog post discussing performance. However, performance is a difficult topic to discuss because it is extremely workload dependent. No matter what I or anyone says, or any benchmarks you see, the best measure of a storage stack's performance will ultimately be a test of your own workloads.

Re: The State of ZFS on Linux

#118

Post by ryao »

foobarqux wrote: post #34
ryao wrote: post #2

I am the author. Feel free to respond with questions. I will be watching for questions throughout the day.

Can you talk about performance of virtual machine disks on ZFS? How is ZFS better/worse than BTRFS?

Before I answer that, let me say that ZFS has two options for the storage of virtual disk for virtual machines. The first is the traditional file. The second is the zvol. The zvol is a virtual block device that is a lower overhead option than a traditional file. The default internal records (blocks) vary for each. On datasets where files are stored, this is called the recordsize and it is 128KB by default. This is a per file property that is set with the value of the dataset's recordsize at the time of creation. On zvols, the volblocksize is 8KB by default. This is set at the time of creation of the zvol. In both cases, partial block writes cause a read-modify-write penalty and typically, zvols are more performant by default. At the same time, LZ4 compression seems to have the counterintuitive consequence of making the larger recordsize about equal in performance in filebench tests that I have done, so it is hard to say which is ultimately better. It is also worth noting that there are several improvements in the pipeline for ZoL zvols that should improve its performance:

https://github.com/zfsonlinux/zfs/pull/2484

With that introduction out of the way, the actual answer to your first question is that it is very dependent on your workload, so I cannot provide a solid answer, but I can discuss my own personal experience. I am involved with ZoL because I was interested in the performance of virtual storage for a home server in 2011. The technology at the time could not compare to ZoL and as far as I know, still cannot compare. In specific, I had 6x Samsung HD204UI drives connected to an AMD Phenom X6 1090T. A configuration with MD RAID 6 + LVM + ext4 did not manage more than 20MB/sec, regardless of whether I used KVM or Xen. A raidz2 configuration using ZFSOnLinux did 220MB/sec. someone with 4 disk recently had a similar experience about a week ago where MD RAID 5 + LVM + XFS and could not get more than 44MB/sec while a ZFS raidz1 configuration managed 210MB/sec if I recall correctly.

As for how ZFS is better/worse than btrfs, ZFS has several advantages in terms of its implementation. In specific, it has ARC that provides a scan resistant cache to maintain performance consistent. It has L2ARC for using flash to extend that cache. It has the ZFS Intent Log, which allows it to avoid blocking on expensive full merkle tree updates. This has allowed ZFS to outperform btrfs in ways that amazed the btrfs developers:

http://comments.gmane.org/gmane.comp.file-systems.btrfs/2754...

It has SLOG devices to allow flash to be used to accelerate ZIL. It also has a custom IO elevator that does a very good job of ensuring performance consistency:

https://twitter.com/lmarsden/status/383938538104184832/photo...

Quite honestly, here are the 5 hypothical areas in terms of where performance can be in any given comparison and what I expect the distribution to be:

1. Areas where ZFS significant outperforms btrfs. I expect there are many of these.

2. Areas where ZFS slightly outperforms btrfs. I expect that there are many of these.

3. Areas where ZFS and btrfs are equal. I expect that there are some of these.

4. Areas where btrfs slightly outperforms ZFS. I think some of these probably exist.

5. Areas where btrfs significant outperforms ZFS. I do not expect any of these to exist. If they do, they indicate bugs in the ZFS kernel driver that need to be corrected.

The areas where I think btrfs might significantly outperform ZFS today are:

1. Uncached directory lookups (getdents performance). ZFS does not currently have directory prefetching and btrfs might. This only affects cold cache performance, so it does not affect production usage and has been a low priority. It will likely be fixed in the next 12 months.

2. Small file performance. btrfs does block suballocation while ZFS does not. This should change in 0.6.4 when ZFS will begin storing small files in the dnode (the ZFS equivalent of the inode).

That said, there is nothing I or anyone can say that is a valid substitute for your own testing and I encourage you to run your own tests.

I hope that this answers your question.

Re: The State of ZFS on Linux

#119

Post by ryao »

Andys wrote: post #74

I used ZFSonLinux on my laptop and workstation for a couple of years now, with Ubuntu, without any major problems. When I tried to use it in production, I didn't get data loss but I hit problems: * Upgrading is a crapshoot: Twice, it failed to remount the pool after rebooting, and needed manual intervention. * Complete pool lockup: in an earlier version, the pool hung and I had to reboot to get access to it again. If…

I mentioned most of these issues in the supplementary blog posts. Here is where each stands:

* There are issues when upgrading because the initramfs can store an old copy of the kernel module and the /dev/zfs interface is not stabilized. This will be addressed in the next 6 months by a combination of two things. The first is /dev/zfs stabilization. The second is bootloader support for dynamic generation of initramfs archives. syslinux does this, but it does not at this time support ZFS. I will be sending Peter Alvin patches to add ZFS support to syslinux later this year. Systems using the patched syslinux will be immune to this problem while systems using GRUB2 will likely need to rely on the /dev/zfs stabilization.

* There are many people who do not have problems, but this is certainly possible. Much of the weirdness should be fixed in 0.6.4. In particular, I seem to have fixed a major cause of rare weirdness in the following pull requests, which had the side benefit of dramatically increasing performance in certain workloads:

https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/zfs/pull/2411

* The above pull requests have a fairly dramatic impact on NFS performance. Benchmarks shown to me by SoftNAS indicate that all performance metrics have increased anywhere from 1.5 to 3 times. Those patches have not yet been merged as I need to address a few minor concerns from the project lead, but those will be rectified in time for 0.6.4. Additional benchmarks by SoftNAS have shown that the following patch that was recently merged increases performance another 5% to 10% and has a fairly dramatic effect on CPU utilization:

https://github.com/zfsonlinux/zfs/commit/cd3939c5f06945a3883...

* There is opportunity for improvement in this area, but it is hard for me to tell what you mean. In particular, I am not certain if you mean minimum latency, maximum latency, average latency or the distribution of latency. In the latter case, the following might be relevant:

https://twitter.com/lmarsden/status/383938538104184832/photo...

That said, I believe that the kmem patches that I linked above will also have a positive impact on SSDs. They reduce contention in critical code paths that affect low latency devices.

Additionally, there is at least one opportunity to improve our latencies. In particular, ZIL could be modified to use Force Unit Access instead of flushes. The problem with this is that not all devices honor Force Unit Access, so making this change could result in data loss. It might be possible to safely make it on SLOG devices as I am not aware of any flash devices that disobey Force Unit Access. However, data integrity takes priority. You can test whether a SLOG device would make a difference in latencies by setting sync=disabled temporarily for the duration of your test. All improvements in the area of SLOG devices will converge toward the performance of sync=disabled. If sync=disabled does not improve things, the bottleneck is somewhere else.

* These alternatives operate on the block device level and add opportunity for bugs to cause cache coherence problems that are damaging to a filesystem on top. They are also unaware of what is being stored, so they cannot attain the same level of performance as a solution that operates on internal objects.

Re: The State of ZFS on Linux

#120

Post by ryao »

voltagex_ wrote: post #89
ryao wrote: post #52

Earlier quoted context omitted.

Below are a few remarks: To expand on your LZ4 comment, LZ4 is 3 times faster than LZO at decompression: https://code.google.com/p/lz4/ However, LZ4's real innovation is on incompressible data. I do not have a reference at the moment, but there have been benchmarks on the userland lz4 tool on incompressible data that show it processing 10GB/sec. This is because LZ4 uses a hash table that enables it to give up when tr…

Is it possible to integrate the ZoL modules with DKMS so that kernel upgrades are handled automagically?

Yes. It is done this way on EPEL distributions (i.e. Fedora, CentOS, Scientific Linux) and Ubuntu. However, it is not presently done on Debian and Gentoo. I am hopeful that this will change on Debian in the future. Gentoo makes kernel upgrades the responsibility of the system administrator, so this does not really apply to it.

WRT to Gentoo, Gentoo does have a tool called genkernel that I consider to make such upgrades fairly easy. It can handle any of initramfs generation, kernel compilation, kernel configuration and bootloader configuration (GRUB2). However, it does have a moderate learning curve. I am also one of the genkernel developers, so I likely have an implicit advantage over users in understanding genkernel. In particular, I find some users have difficulty discovering its functionality, despite it having things fairly well documented in the man page. There is likely more that needs to be done in this area.

Post reply on HN