Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

ilsistemista.net

Post reply on HN
Print view View original

51–60 of 85 posts

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#51

Post by ryao »

specialist wrote: post #45
tete wrote: post #42

Here a few things to note: BTRFS has had Copy on Write disabled, while ZFS (not possible, cause the whole idea of the FS is intended to be copy on write), which actually makes BTRFS look even worse compared to ZFS, cause BTRFS writes once instead of twice and has most of its features not work, while still performance pretty bad. But then it's also younger. Both ZFS and BTRFS (only know specifics of ZFS) can be config…

Is there something like Aphyr's Jespen for file systems? https://aphyr.com/tags/Jepsen I've been curious about ZFS, btrfs, etc. But as a layperson, I don't have the technical chops, gumption, wherewithal to figure what's what. Reading posts (comments) about the edge cases where they fail (data, performance, missing features) leaves me more baffled.

There is plenty there to read. I would rather not spend my Sunday reading that to see if I can think of something equivalent. However, you might find these pages helpful:

http://open-zfs.org/wiki/System_Administration http://open-zfs.org/wiki/Performance_tuning http://open-zfs.org/wiki/Hardware

There are also mailing lists and IRC channels where you can ask questions. On freenode, there are #zfsonlinux, #openzfs, #openzfs-osx and #zfs.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#53

Post by specialist »

ryao wrote: post #51
specialist wrote: post #45

Earlier quoted context omitted.

Is there something like Aphyr's Jespen for file systems? https://aphyr.com/tags/Jepsen I've been curious about ZFS, btrfs, etc. But as a layperson, I don't have the technical chops, gumption, wherewithal to figure what's what. Reading posts (comments) about the edge cases where they fail (data, performance, missing features) leaves me more baffled.

There is plenty there to read. I would rather not spend my Sunday reading that to see if I can think of something equivalent. However, you might find these pages helpful: http://open-zfs.org/wiki/System_Administration http://open-zfs.org/wiki/Performance_tuning http://open-zfs.org/wiki/Hardware There are also mailing lists and IRC channels where you can ask questions. On freenode, there are #zfsonlinux, #openzfs, #op…

Sorry, I didn't mean for you to read it, just linked to be clear. The Jespen effort verifies the reliability claims of various persistence schemes. Independent verification, if you will.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#54

Post by cmurf »

SXX wrote: post #24
josefbacik wrote: post #20

Earlier quoted context omitted.

There are no docs. I'm not a qcow2 expert, what I know is very basic so anything I say about qcow2 can be very wrong. So qcow has a read only base image that gets updated when we change things. The image format just had the changes from the original image. So you update a package, it adds some metadata to point at the new stuff and adds the data in and you are done. So with btrfs you have this image on top of btrfs,…

I tested it with preallocated raw images on Btrfs and that setup was slow as well. If I get you right it's must be possible to somehow allign VM disk sectors to Btrfs sectors for performance boost. Or compression and deduplication usage going to kill performance anyway?

Btrfs internally use logical addresses for all extents. The making from logical to physical is done via the chunk tree which not only indicates physical sector but also the device. So the reference for a file extent says nothing about what device the extent is on or the replication (raid profile) since that is all a function of the chunk and dev trees.

I think Btrfs for a guest F's is best pointed to an LV, rather than qcow2. It's been awhile since I benchmarked that compared to 'qemu-img create -f qcow2 -o nocow=on' which will set xattr +C on the file making it nocow. The nocow xattr helps a lot with this problem.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#56

Post by ryao »

tete wrote: post #42

Here a few things to note: BTRFS has had Copy on Write disabled, while ZFS (not possible, cause the whole idea of the FS is intended to be copy on write), which actually makes BTRFS look even worse compared to ZFS, cause BTRFS writes once instead of twice and has most of its features not work, while still performance pretty bad. But then it's also younger. Both ZFS and BTRFS (only know specifics of ZFS) can be config…

There is a UFS driver for Linux, but it is a reimplementation rather than a port, so it's performance numbers would not be comparable. Also, it attempts to support many variants of UFS rather than just one:

https://github.com/torvalds/linux/blob/master/Documentation/...

It does not support the latest UFS developments in FreeBSD, NetBSD, etcetera, so its performance is also limited by the older disk format versus the newer formats used by drivers on other platforms.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#57

Post by qaq »

josefbacik wrote: post #23
loeg wrote: post #22

Earlier quoted context omitted.

Shouldn't ZFS have similar problems with overwrite workloads?

ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.

It's not just years it's the types of deployments. ZFS backs a very large number of very large critical systems.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#58

Post by ryao »

specialist wrote: post #53
ryao wrote: post #51

Earlier quoted context omitted.

There is plenty there to read. I would rather not spend my Sunday reading that to see if I can think of something equivalent. However, you might find these pages helpful: http://open-zfs.org/wiki/System_Administration http://open-zfs.org/wiki/Performance_tuning http://open-zfs.org/wiki/Hardware There are also mailing lists and IRC channels where you can ask questions. On freenode, there are #zfsonlinux, #openzfs, #op…

Sorry, I didn't mean for you to read it, just linked to be clear. The Jespen effort verifies the reliability claims of various persistence schemes. Independent verification, if you will.

ZFS has a utility called ztest that performs stochastic testing on the core ZFS code in userspace with assertions enabled. It catches many reliability issues and is something that anyone can run. There is also the zinject framework for doing fault injection.

As for independent verification, here are the top two results from searching google from my IP address for "Zfs paper reliability":

http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf http://research.cs.wisc.edu/adsl/Publications/zfs-corruption...

There is nothing surprising to me there. The merkle tree with good quality 256-bit checksums strongly guarantees that corruptions are detected and redundancy allows for correction.

The main ways of messing up would be a buggy driver, something else in the kernel damaging the driver's data structures or hardware corrupting memory. The stochastic testing gives some protection against the first of those. I guess you could target the two things that live outside of the merkle tree (labels and ZIL blocks), but those are self checksummed and used in ways that minimize the potential for issues.

Here is a summary of data integrity features that I wrote:

http://open-zfs.org/w/index.php?title=System_Administration#...

It is possible that I missed some of the more obscure protections.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#59

Post by ryao »

cmurf wrote: post #54
SXX wrote: post #24

Earlier quoted context omitted.

I tested it with preallocated raw images on Btrfs and that setup was slow as well. If I get you right it's must be possible to somehow allign VM disk sectors to Btrfs sectors for performance boost. Or compression and deduplication usage going to kill performance anyway?

Btrfs internally use logical addresses for all extents. The making from logical to physical is done via the chunk tree which not only indicates physical sector but also the device. So the reference for a file extent says nothing about what device the extent is on or the replication (raid profile) since that is all a function of the chunk and dev trees. I think Btrfs for a guest F's is best pointed to an LV, rather th…

ZFS also does better when the guests are stored on zvols. If volblocksize=4K is set on creation, it can avoid read-modify-write overhead. Regular files can be used too, but those are somewhat higher overhead. In that case, recordsize=4K can be used.

That said, I do not think the tests involved nesting CoW file systems.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#60

Post by rurban »

josefbacik wrote: post #23
loeg wrote: post #22

Earlier quoted context omitted.

Shouldn't ZFS have similar problems with overwrite workloads?

ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.

5x more core ZFS contributors sounds to me worse and not better than BTRFS. The more the worse usually.
Post reply on HN