Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of ZFS on Linux

clusterhq.com

Post reply on HN
Print view View original

101–110 of 125 posts

Re: The State of ZFS on Linux

#101

Post by colin_mccabe »

ryao wrote: post #57
foobarqux wrote: post #33

Earlier quoted context omitted.

Can you speak more about why ZFS is better than BTRFS?

A btrfs versus ZFS comparison probably deserves a blog post of its own, but I will try to address your question. I wrote the following on this topic last year: https://groups.google.com/d/msg/funtoo-dev/g9OY_vqVpCM/VTKF8... However, significant time has passed and it requires some corrections to be current: 1. I have not heard of any recent data corruption issues in btrfs, although I have not looked into them lately.…

As others have commented, CRC32 or CRC64 being "weak" in a cryptographic sense doesn't mean it's not suitable for detecting disk errors. x86 has a crc32 instruction, making it far faster than any other option. By the way, TCP doesn't use CRC32... it uses a simple additive checksum.

Chris Mason wrote most of btrfs while he was at Oracle. Hence, the GPLv2 grants an implicit patent license to whatever patents Oracle might hold that bear on btrfs.

Re: The State of ZFS on Linux

#102

Post by xioxox »

barrkel wrote: post #29
xioxox wrote: post #22

Earlier quoted context omitted.

I've setup btrfs on software mdraid (raid 6) as a backup system (not the only backup!). You still get the checksums and snapshotting, but not the flexibility of the btrfs raid system. It has the advantage of being easy to grow, unlike zfs, which can't be resized once created. We've encountered no problems, even though it has been running for around three years of rsyncing and snapshotting.

zfs can grow - new vdevs can be added, and existing vdevs can have their disks replaced one at a time with larger disks. A bigger downside of ZFS, IMO, is lack of defragmentation and similar larger scale pool management. If you ever push a ZFS pool close to its space limit, you can end up with fragmentation that never really goes away, even if you delete lots of files. The recommended solution is to recreate the pool…

zfs can sort of grow by adding vdevs, as you say, however it's pretty wasteful due to the new parity drives It was much more efficient to expand the mdraid raid 6 and expand the btrfs onto that. The other backup server does use zfs (albeit the user mode fuse version). I set up that system in a similar way putting zfs-fuse on mdraid 6.

Re: The State of ZFS on Linux

#103

Post by Rapzid »

ryao wrote: post #2

I am the author. Feel free to respond with questions. I will be watching for questions throughout the day.

I followed you and Brian's contributions quite closely at my previous job. We had some pretty extreme backup targets for our VPS's and the old tools were starting to become bothersome; particularly for keeping a couple hundred million files sync'd between data centers. I knew about ZFS's send/receive but we were a linux shop.. About the time we(I) were going to make some major changes to the backup systems I gave ZFS another Google, as I like to re-check my assumptions every now and then, and discovered ZoL went "stable" just that month! I immediately pushed to give it a spin and the rest was history. Learning ZFS was fantastic fun. It challenged everything I thought a filesystem was capable off. L2Arc, snapshots(cloning and shared data wut?!), ZVol's, checksum's, and on and on and on. Thanks for all your hard work and making this possible!

Re: The State of ZFS on Linux

#104

Post by DiabloD3 »

phil21 wrote: post #100
DiabloD3 wrote: post #80

Earlier quoted context omitted.

Yes and no. The optimal number for RAIDZ/Z1/Z2, claimed by many benchmarks, is (power of two count of drives) plus (number of drives for parity)[1], with the upper bound being the likelihood of another drive failing before rebuild can complete. There is a tradeoff of, for example, using 3 RAIDZs each with a third of the drives in your pool, or 1 RAIDZ3 using all the drives in your pool. In the first case, a rebuild t…

You're pretty wrong regarding performance here. In ZFS, performance is directly bounded by the number of vdevs given to a pool. If you make one 90 drive vdev, you are more or less limiting yourself to the iops of a single disk. It's a bit more complicated than that, as your throughput increases. At work where we use ZFS extensively we more or less use # of vdevs * I/O performance of a single drive = total "worst-case…

I actually mentioned that, but maybe not as well as I should have. I did say 8, 12, and 16 drives in an array will perform similarly.

However, in cases where you are using SSDs for ZIL/L2ARC, vdev spam isn't as big of a factor. In the example I gave I was illustrating the amount of usable storage you could get in the context of having it reliable, not comparing performance of RAID-Z vs non-ZFS RAID.

Re: The State of ZFS on Linux

#105

Post by DiabloD3 »

lmz wrote: post #87
DiabloD3 wrote: post #81

Earlier quoted context omitted.

Honestly, I recommend XFS over ext4. It seems to be a much more mature file system, and Redhat-family distros (RHEL, Centos, Scientific, etc) have switched to XFS as the default filesystem (instead of moving to ext4 from 6.x's default of ext3; 6.x does not support XFS or ext4 for root). XFS performs better out of the box on a wide range of hardware, while seemingly giving stronger data reliability guarantees (but not…

RHEL6 uses ext4 as default[1] filesystem. It certainly does support ext4 as root. [1]: https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...

Weird, I had to install RHEL6 for a customer, and it defaulted to ext3 and ext4 was not selectable.

Re: The State of ZFS on Linux

#106

Post by ryao »

colin_mccabe wrote: post #101
ryao wrote: post #57

Earlier quoted context omitted.

A btrfs versus ZFS comparison probably deserves a blog post of its own, but I will try to address your question. I wrote the following on this topic last year: https://groups.google.com/d/msg/funtoo-dev/g9OY_vqVpCM/VTKF8... However, significant time has passed and it requires some corrections to be current: 1. I have not heard of any recent data corruption issues in btrfs, although I have not looked into them lately.…

As others have commented, CRC32 or CRC64 being "weak" in a cryptographic sense doesn't mean it's not suitable for detecting disk errors. x86 has a crc32 instruction, making it far faster than any other option. By the way, TCP doesn't use CRC32... it uses a simple additive checksum. Chris Mason wrote most of btrfs while he was at Oracle. Hence, the GPLv2 grants an implicit patent license to whatever patents Oracle mig…

Saying that a checksum is weak in a cryptographic sense would mean that it is possible for a malicious intelligence to generate collisions. This is not what I meant when I said that CRC was weak. Instead, I meant that it is trivial for ordinary glitches to corrupt data in ways that the checksums fail to detect. Things byte swaps and non-adjacent double bit flips are all that is necessary. The link I provided elaborates on this:

http://noahdavids.org/self_published/CRC_and_checksum.html

As for patent grants, you are thinking of the GPLv3. The GPLv2 does not provide any patent grant. That is one of the reasons why the GPLv3 was made. If you place code under the GPLv2, you do not necessarily provide a grant to any software patents to which you have rights that cover it. People who wish to use code encumbered by such patents must license it or be at risk a lawsuit.

Re: The State of ZFS on Linux

#107

Post by ryao »

cmurf wrote: post #98
ryao wrote: post #57

Earlier quoted context omitted.

A btrfs versus ZFS comparison probably deserves a blog post of its own, but I will try to address your question. I wrote the following on this topic last year: https://groups.google.com/d/msg/funtoo-dev/g9OY_vqVpCM/VTKF8... However, significant time has passed and it requires some corrections to be current: 1. I have not heard of any recent data corruption issues in btrfs, although I have not looked into them lately.…

Metadata/data checksums are CRC-32C on all platforms and is per 4KB fs block. And ext4's (optional) checksumming also uses it. TCP is 16-bit and not CRC, although ethernet makes use of CRC-32. While SHA-2 is a cryptographic function and CRC-32C is not, and therefore inferior, as a checksum in the context of mostly but not entirely trusted hardware is adequate. It's also fast to the degree on modern hardware there's n…

You are correct. The ethernet frames use CRC32 while the IP frames use 16-bit CRC. While the checksums need not be of cryptographic quality (and not all checksum algorithms in ZFS are), there are multiple trivial ways in which the wrong data can match a CRC32 checksum and that is inadequate for the purposes of data integrity. It does help, but it does not go far enough.

As for my assertion that btrfs uses 64-bit CRC on 64-bit processors, this was the result of a misunderstanding between myself and the btrfs developers in #btrfs on freenode last year. I double checked with them and it turns out that you are correct that btrfs uses 32-bit CRC even on 64-bit processors. That makes btrfs much worse in this area than I previously thought.

Thanks for clearing some of the confusion about btrfs send/receive on its snapshots. To be clear, does this mean that btrfs will refuse to send a "writeable snapshot"?

Re: The State of ZFS on Linux

#108

Post by ryao »

turrini wrote: post #67

I've created the script below a while (year) ago. It (deb)bootstrap a working Debian Wheezy with ZFS on root (rpool) using only 3 partitions: /boot(128M) swap(calculated automatically) rpool(according to # of your disks, mirrored or raidz'ed). All commentaries are in Brazilian Portuguese. I didn't have time to translate it to English. Someone could do it and fill a push request. https://github.com/turrini/scripts/blo…

Thanks for sharing. I will let Debian users interested in / on ZFS know that this is available as they ask me about this sort of thing.

Re: The State of ZFS on Linux

#109

Post by mbreese »

ryao wrote: post #96
mbreese wrote: post #53

Earlier quoted context omitted.

Basically I mean integration with the kernel's code base and all of the testing that entails. After their initial development, file systems all end up migrating to the kernel's code base. So, I'm not thinking in terms of technical API support, but more development/testing/integration support.

That is not a requirement. ZoL has the most sophisticated build system of any Linux kernel module in the world to enable it to live outside of the main tree. ZoL relies on autotools' API checks to do this. In addition, the project has an automated buildbot that helps us to detect regressions in pull requests before they are merged. It is similar in principle to how lustre filesystem development is done. Lustre is als…

You know... I hadn't given much thought to Lustre. And that is a very good comparison. Lustre is completely out of kernel (in terms of development), so it's not like you're the first to try this.

I guess it really depends on where you want ZoL to be deployed. Lustre is typically deployed on HPC clusters and is managed by people who understand it, how to set it up, and how to manage it. It's not a trivial system to get working, and requires a pretty large budget. It just isn't setup on your typical server. Lustre also has a few big names behind it to provide development resources and support (Whamclound, now Intel).

What is the target market for ZoL? Do you want it to work on servers? Clusters? Personal workstations? They are very different markets.

I'm using ZFS right now for a single storage server that doesn't need to support a large cluster, so ZFS/NFS works great. But I'm using FreeBSD. For my Linux servers, I wouldn't think of using a filesystem that wasn't natively supported by Red Hat. I just don't want the management headache. I'm okay running a single FreeBSD box for my storage needs, but I'm fearful of what happens when I'd need to scale things out to multiple servers.

I wish you luck, I really do. It seems like you're not going into this blind and you're making good decisions. But it will be an uphill battle and you'll always have that specter of Oracle looming over you. It doesn't matter if you're technically/legally right on the licensing front - you'll still have it looming over you until Larry signs off on it (like you said below).

If you can get that sign off, then all bets are off, and you'd be golden!

Re: The State of ZFS on Linux

#110

Post by mbreese »

ryao wrote: post #92
nailer wrote: post #37

Putting production data on a driver maintained outside the mainline Linux kernel is a bad idea. That isn't a licensing argument - I'm happy to use a proprietary nvidia.ko for gaming tasks, for example, because I won't be screwing up anyone's data if it breaks.

You could be "screwing up" someone's data if an in-tree filesystem breaks. If you read the supplementary blog posts, you would have seen the following: http://lwn.net/Articles/437284/ Nearly all in-tree filesystems can fail in the same way described there. ZFS cannot. That being said, no filesystem is a replacement for backups. This applies whether you use ZFS or not. If you care about your data, you should have back…

Well, to be honest, all filesystems that are currently in the kernel tree started out as being maintained outside the tree. Inclusion into the kernel is normally one of the end goals. It's part of the standard progression - 1) rapid development outside of the tree, 2) once the filesystem is stable, it negotiates for inclusion, 3) inclusion into the kernel, 4) maintenance / updates as part of the main kernel development process.

ZoL isn't even trying to get included into the kernel, so it's a bit of an odd duck here.

Post reply on HN