Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of ZFS on Linux

clusterhq.com

Post reply on HN
Print view View original

51–60 of 125 posts

Re: The State of ZFS on Linux

#51

Post by ryao »

astral303 wrote: post #50

Tried using ZFS in the earnest and got spooked, felt it was not production ready. Wanted to use ZFS for MongoDB on Amazon Linux (primarily for compression, but also for snapshot functionality for backups). Tried 0.6.2. Ended up running into a situation where a snapshot delete hung and none of my ZFS commands were returning. The snapshot delete was not killable with kill -9. https://github.com/zfsonlinux/zfs/issues/12…

Your deadlock was likely caused by the sole regression to get by us in the 0.6.2 release:

https://github.com/zfsonlinux/zfs/commit/a117a6d66e5cf1e9d4f...

This occurred because it was rare enough that neither us nor the buildbots caught it back in Feburary. George Wilson wrote a fix for it in Illumos rather promptly. However, Illumos and ZoL projects had different formats for the commit titles of regression fixes. In specific, the Illumos developers would reuse the same exact title while the ZoL developers would generally expect a different title, so we missed it when merging work done in Illumos. I caught it in November when I was certain that George had made a mistake and noticed that our code and the Illumos code was different. It is fixed in 0.6.3. The fix was backported to a few distribution repositories, but not to all of them.

The 0.6.3 release was notable for having a very long development cycle. As I described in the blog post, the project will begin doing official bug fix releases when 1.0 is tagged. That should ensure that these fixes become available to all distributions much sooner. In the mean time, future releases are planned to have much shorter development cycles than 0.6.3 had, so fixes like this will become available more quickly.

That being said, I was at the MongoDB office in NYC earlier this year to troubleshoot poor performance on MongoDB. I will refrain from naming the MongoDB developer with whom I worked lest he become flooded with emails, but my general understanding is that 0.6.3 resolved the performance issues that the MongoDB had observed. Future releases should further increase performance.

Re: The State of ZFS on Linux

#52

Post by ryao »

nisa wrote: post #47
foobarqux wrote: post #33

Earlier quoted context omitted.

Can you speak more about why ZFS is better than BTRFS?

I'm not the op but I'm running ZoL on a small cluster with 200 HDDs and 50 machines. I've also tested btrfs. Why is ZFS better (IMHO): + Tooling. The zpool and zfs commands are clear, easy to use and well documented. btrfs e.g. has no way to get a list of files with checksum problems. Also automounting and setting flexible mount points as well as the whole zfs get/set concept felt like a really concise way to configu…

Below are a few remarks:

To expand on your LZ4 comment, LZ4 is 3 times faster than LZO at decompression:

https://code.google.com/p/lz4/

However, LZ4's real innovation is on incompressible data. I do not have a reference at the moment, but there have been benchmarks on the userland lz4 tool on incompressible data that show it processing 10GB/sec. This is because LZ4 uses a hash table that enables it to give up when trying to compress data far sooner than other lempel-ziv implementations do.

As for stability, it is hard to for me to tell what you mean by that, but I just wrote a series of blog posts on the topic. If there are any outstanding issues not listed there, please file an issue on github:

https://github.com/zfsonlinux/zfs/issues/new

I also encourage you to file an issue requesting support for cgroup bsaed IO throttling. You are the first person to mention it.

As for not compiling with new kernels, the 0.6.3 release supports Linux 2.6.26 to Linux 3.16 while HEAD adds support for Linux 3.17 release candidates.

I briefly touched on the rootfs on ZFS issue in my blog post. I plan to add ZFS support to syslinux in the near future as per a discussion that I had with Peter Alvin at LinuxCon North America 2014. I expect this to resolve the issue for those willing to use syslinux as their bootloader.

Re: The State of ZFS on Linux

#53

Post by mbreese »

michael_h wrote: post #43
mbreese wrote: post #41

I love ZFS, and I love working with Linux, but I can't help but worry about using ZFS on Linux. Without the needed support from the kernel side, I don't see how it can be useful for production. I can see using it on personal workstations, but for any situation where data loss is critical, you just won't see any uptake. Because of the licensing, ZFS can never be anything more than a second-class citizen on Linux. That…

Without the needed support from the kernel side Can you clarify what you mean by that?

Basically I mean integration with the kernel's code base and all of the testing that entails. After their initial development, file systems all end up migrating to the kernel's code base.

So, I'm not thinking in terms of technical API support, but more development/testing/integration support.

Re: The State of ZFS on Linux

#54

Post by astral303 »

ryao wrote: post #51
astral303 wrote: post #50

Tried using ZFS in the earnest and got spooked, felt it was not production ready. Wanted to use ZFS for MongoDB on Amazon Linux (primarily for compression, but also for snapshot functionality for backups). Tried 0.6.2. Ended up running into a situation where a snapshot delete hung and none of my ZFS commands were returning. The snapshot delete was not killable with kill -9. https://github.com/zfsonlinux/zfs/issues/12…

Your deadlock was likely caused by the sole regression to get by us in the 0.6.2 release: https://github.com/zfsonlinux/zfs/commit/a117a6d66e5cf1e9d4f... This occurred because it was rare enough that neither us nor the buildbots caught it back in Feburary. George Wilson wrote a fix for it in Illumos rather promptly. However, Illumos and ZoL projects had different formats for the commit titles of regression fixes. In…

Thank you so much for the information! This is very encouraging. I will definitely give 0.6.3 a whirl!

Re: The State of ZFS on Linux

#55

Post by jaytaylor »

foobarqux wrote: post #33
DiabloD3 wrote: post #25

I've used ZoL since it was created, and zfs-fuse before that. I ran it on my workstation for a few years (managing a 4x750gb RAID-Z (= ZFS's RAID-5 impl), with ext3 on mdadm RAID 1 2x400gb root), and then swapped to BTRFS for 2x2TB BTRFS native RAID 1 (which was Oracle's ZFS competitor that seems to be largely abandoned although I see commits in the kernel changelog periodically), and now back to ZFS on a dedicated f…

Can you speak more about why ZFS is better than BTRFS?

BTRFS is still maturing. Kernel bugs involving BTRFS are still popping up pretty regularly.

Re: The State of ZFS on Linux

#56

Post by [deleted] »

foobarqux wrote: post #33
DiabloD3 wrote: post #25

I've used ZoL since it was created, and zfs-fuse before that. I ran it on my workstation for a few years (managing a 4x750gb RAID-Z (= ZFS's RAID-5 impl), with ext3 on mdadm RAID 1 2x400gb root), and then swapped to BTRFS for 2x2TB BTRFS native RAID 1 (which was Oracle's ZFS competitor that seems to be largely abandoned although I see commits in the kernel changelog periodically), and now back to ZFS on a dedicated f…

Can you speak more about why ZFS is better than BTRFS?

[deleted]

Re: The State of ZFS on Linux

#57

Post by ryao »

foobarqux wrote: post #33
DiabloD3 wrote: post #25

I've used ZoL since it was created, and zfs-fuse before that. I ran it on my workstation for a few years (managing a 4x750gb RAID-Z (= ZFS's RAID-5 impl), with ext3 on mdadm RAID 1 2x400gb root), and then swapped to BTRFS for 2x2TB BTRFS native RAID 1 (which was Oracle's ZFS competitor that seems to be largely abandoned although I see commits in the kernel changelog periodically), and now back to ZFS on a dedicated f…

Can you speak more about why ZFS is better than BTRFS?

A btrfs versus ZFS comparison probably deserves a blog post of its own, but I will try to address your question. I wrote the following on this topic last year:

https://groups.google.com/d/msg/funtoo-dev/g9OY_vqVpCM/VTKF8...

However, significant time has passed and it requires some corrections to be current:

1. I have not heard of any recent data corruption issues in btrfs, although I have not looked into them lately.

2. btrfs now has experimental RAID 5/6 support, but is neither production ready nor as refined as ZFS' raidz.

3. I should have said "inline block-based data deduplication". You can (ab)use reflinks to achieve a file-level data deduplication in btrfs, but it is not quite the same. btrfs now has a bedup tool that makes using reflinks somewhat easier now:

https://btrfs.wiki.kernel.org/index.php/Deduplication

4. btrfs now has some kind of incremental send/recv operation. However, it is not clear to me how it handles consistency issues from having "write-able snapshots":

https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

5. Illumos' ZFS implementation is now able to store small files in the dnode, which is improves its efficiency when storing small files in a manner similar to btrfs' block suballocation. This feature will likely be in ZoL 0.6.4.

Aside from those corrections, what I wrote in that mailing list email should still be relevant today. However, there are a few advantages that ZFS has over btrfs that I recall offhand that I do not see here there or in nisa's reply:

0. ZFS uses 256-bit checksums with algorithms that are still considered to be good today. btrfs uses checksum algorithms that are known to be weak. In specific, btrfs uses CRC32 on 32-bit processors and CRC64 on 64-bit processors. CRC32 is the same algorithm used by TCP/IP. Its deficiencies are well documented:

http://noahdavids.org/self_published/CRC_and_checksum.html

I have not examined CRC64, but I am not particularly confident in it. btrfs should have room in its on-disk data structures that would allow it to implement 256-bit checksums in a future disk format extension, but until then, its checksum implementations are vastly inferior.

1. The ztest utility that I described in the blog allows ZFS developers to catch issues that would have otherwise gone into production and debug them from userland. No other filesystem has something like quite like it.

2. ZFSOnLinux is the only kernel filesystem driver that is kernel version-independent, so if you are unable to upgrade your kernel, you can still get fixes. The inability of people to always update their kernels is an issue Linus mentioned at LinuxCon North America 2014.

3. The CDDL gives the ZFSOnLinux a patent grant for the ZFS patent portfolio. This is something that btrfs does not have and will likely never have unless Oracle decides to provide one. Consequently, Oracle is the only company in the world that I know is able to ship products incorporating the btrfs source code without being at risk should btrfs infringe on one of the dozens if not hundreds of patents in the ZFS patent portfolio. A small subset of them can be accessed from the Jeff Bonwick Wikipedia page:

https://en.wikipedia.org/wiki/Jeff_Bonwick

Re: The State of ZFS on Linux

#58

Post by ashayh »

ZFS, and most* other file systems are all about _one_ computer system.

While ZFS data integrity features may be useful, they don't prevent the wide variety of things that can go wrong on a _single_ computer. You still need site redundancy, multiple physical copies, recovery from user errors etc.

Large, modern enterprises are better off keeping data on application layer "filesystems" or databases, since they can more easily aggregate the storage of hundreds or thousands of physical nodes. ZFS doesn't help with anything special here.

For the average home user, ZoL modules are a hassle to maintain. You are better of setting up FeeNAS on a 2nd computer if you really want to use ZFS. Otherwise there is nothing much over what XFS, EXT4 or btrfs can offer.

The 'ssm' set of tools to manage LVM, and other built in file systems, is more easier for home users with regular needs.

GlusterFS and others are distributed file systems, but suffers from additional complexity at the OS and management layer.

Re: The State of ZFS on Linux

#59

Post by nisa »

ryao wrote: post #52
nisa wrote: post #47

Earlier quoted context omitted.

I'm not the op but I'm running ZoL on a small cluster with 200 HDDs and 50 machines. I've also tested btrfs. Why is ZFS better (IMHO): + Tooling. The zpool and zfs commands are clear, easy to use and well documented. btrfs e.g. has no way to get a list of files with checksum problems. Also automounting and setting flexible mount points as well as the whole zfs get/set concept felt like a really concise way to configu…

Below are a few remarks: To expand on your LZ4 comment, LZ4 is 3 times faster than LZO at decompression: https://code.google.com/p/lz4/ However, LZ4's real innovation is on incompressible data. I do not have a reference at the moment, but there have been benchmarks on the userland lz4 tool on incompressible data that show it processing 10GB/sec. This is because LZ4 uses a hash table that enables it to give up when tr…

Wow! Thanks for your reply. I've experienced some minor issues with severe broken disks where the SPL layer discards the disk but the zpool is still online. It's not a major issue and I've yet to encounter the issue again I've already filed a related bug: https://github.com/zfsonlinux/zfs/issues/2508 but I'm not an expert. If I can gather more data I'll try to submit as much data and hints I can find.

As for cgroups blkio - there is already a bug report: https://github.com/zfsonlinux/zfs/issues/1952

Sorry. It was not my intention to suggest that ZFS does not compile on new kernel versions. I've never had problems with that. I just wanted to point out that it's not in the mainline kernel and for some people that might be important.

Thanks for your great work on ZoL! It made my life a lot easier :)

Re: The State of ZFS on Linux

#60

Post by DiabloD3 »

foobarqux wrote: post #33
DiabloD3 wrote: post #25

I've used ZoL since it was created, and zfs-fuse before that. I ran it on my workstation for a few years (managing a 4x750gb RAID-Z (= ZFS's RAID-5 impl), with ext3 on mdadm RAID 1 2x400gb root), and then swapped to BTRFS for 2x2TB BTRFS native RAID 1 (which was Oracle's ZFS competitor that seems to be largely abandoned although I see commits in the kernel changelog periodically), and now back to ZFS on a dedicated f…

Can you speak more about why ZFS is better than BTRFS?

Working RAID 5 and 6, RAID "7" (Z3[1], triple parity, where Z/5 is single, and Z2/6 is dual), tiered RAID setups (such as JBOD'ed RAID-Z3s), zvols (as in, but not limited to, swap partitions in ZFS), more nuanced multi-controller and failover/spare setups, write-only journals (ZIL), second tier caching (L2ARC), LZ4 transparent compression, better designed snapshot and snapshot cloning support, more mature CLI tools, support on other OS's (with shared code bases for bonus points), configurable checksum algos, and a few other things that I'm forgetting at the moment.

[1]: Seen that 90 drive Supermicro drive chassis[2]? Three 29-drive RAID-Z3s with 3 hot spares (which is shared across the three RAIDs) in a single storage pool (round robin-esqued), and you could plug that into a 2U with 16 small SSDs (like those Crucial M550 128GBs I use now) for the ZIL/L2ARC farming, and then have, well, near-infinite IO performance. Good luck trying to assemble that with BTRFS.

[2]: A wet dream for CEPH users, too.

Post reply on HN