Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

ilsistemista.net

Post reply on HN
Print view View original

71–80 of 85 posts

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#71

Post by versteegen »

josefbacik wrote: post #21
daurnimator wrote: post #16

Earlier quoted context omitted.

But shouldn't NOCOW mitigate that? (which the article does for some tests)

NOCOW is horribly expensive because we still have to go check and make sure that there are no snapshots pointing at the changing extents. It only solves the fragmentation issue, and if you don't prealloc your image it doesn't even do that.

How about if you have a NOCOW subvolume with just a few files on it, e.g. 3 VM images. Is it still expensive to check for snapshots? Does the cost of checking disappear if you have no snapshots of the volume? As ryao mentioned, you'd still get fragmentation if you have snapshots, so in that case is using NOCOW for a VM image counterproductive?

Seems like the expense of checking could be largely removed with clever enough metadata caching, which probably noone has had time to implement.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#72

Post by ymse »

specialist wrote: post #45
tete wrote: post #42

Here a few things to note: BTRFS has had Copy on Write disabled, while ZFS (not possible, cause the whole idea of the FS is intended to be copy on write), which actually makes BTRFS look even worse compared to ZFS, cause BTRFS writes once instead of twice and has most of its features not work, while still performance pretty bad. But then it's also younger. Both ZFS and BTRFS (only know specifics of ZFS) can be config…

Is there something like Aphyr's Jespen for file systems? https://aphyr.com/tags/Jepsen I've been curious about ZFS, btrfs, etc. But as a layperson, I don't have the technical chops, gumption, wherewithal to figure what's what. Reading posts (comments) about the edge cases where they fail (data, performance, missing features) leaves me more baffled.

Not the same, but there was an article about file system fuzzing recently: https://news.ycombinator.com/item?id=11469535

The presentation is focused on how the fuzzing technique works rather than individual file system performance, but the "time to first bug" is quite telling. Copied here:

    ext4 (2h)
    XFS (1h45)
    GFS2 (8m)
    NTFS (4m)
    NILFS2 (1m)
    HFS (30s)
    HFS+ (25s)
    ReiserFS (25s)
    OCFS2 (15s)
    F2FS (10s)
    BTRFS (5s)
I've had miscellaneous failures with the bottom three, but XFS and ext4 have been rock solid. Unfortunately ZFS and CephFS were not included.

Would love to see a Jepsen-style analysis of the various file systems too.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#73

Post by jwatte »

zbraniecki wrote: post #69
josefbacik wrote: post #7

Earlier quoted context omitted.

Kvm (or any overwrite workload for that matter) is the worst possible workload for btrfs because of COW. We have ideas to address this but honestly it's not high on the list.

Is brtfs a good choice from the power consumption standpoint? I'm on a laptop and trying to maximize my battery life, and am wondering if my choice of brtfs limits it.

You could try nilfs for that.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#74

Post by josefbacik »

MertsA wrote: post #63
ryao wrote: post #46

Earlier quoted context omitted.

Are you sure that disabling CoW solves btrfs' fragmentation issue when you pre-allocate the image? If the volume is snapshotted regularly, CoW should be active once on each extent once until the next snapshot reactivates it. That should mean that the fragmentation would still occur, although more slowly. Is that correct or am I misunderstanding something? I am only superficially familiar with btrfs internals, but I d…

Disabling CoW disables snapshotting.

No it doesn't.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#75

Post by josefbacik »

ryao wrote: post #46
josefbacik wrote: post #21

Earlier quoted context omitted.

NOCOW is horribly expensive because we still have to go check and make sure that there are no snapshots pointing at the changing extents. It only solves the fragmentation issue, and if you don't prealloc your image it doesn't even do that.

Are you sure that disabling CoW solves btrfs' fragmentation issue when you pre-allocate the image? If the volume is snapshotted regularly, CoW should be active once on each extent once until the next snapshot reactivates it. That should mean that the fragmentation would still occur, although more slowly. Is that correct or am I misunderstanding something? I am only superficially familiar with btrfs internals, but I d…

Yeah nocow only works if there are no snapshots, which is why we have to check, so if you use snapshots it falls back to COW.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#76

Post by Already__Taken »

I'd like to see NTFS somewhere in these comparisons just to shake some of that "grass is greener" thinking. The amount I've read on the various file systems I feel like there's a lot on the table there to get out of our hardware by just using a better system. But I've not seen a whole lot about ntfs, because it's all we've got in windows land I suppose.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#78

Post by josefbacik »

versteegen wrote: post #71
josefbacik wrote: post #21

Earlier quoted context omitted.

NOCOW is horribly expensive because we still have to go check and make sure that there are no snapshots pointing at the changing extents. It only solves the fragmentation issue, and if you don't prealloc your image it doesn't even do that.

How about if you have a NOCOW subvolume with just a few files on it, e.g. 3 VM images. Is it still expensive to check for snapshots? Does the cost of checking disappear if you have no snapshots of the volume? As ryao mentioned, you'd still get fragmentation if you have snapshots, so in that case is using NOCOW for a VM image counterproductive? Seems like the expense of checking could be largely removed with clever en…

We have to look up the physical extent in the extent reference tree, so the cost is independent of the number of snapshots and more a function of the fragmentation of the extent tree. The metadata is all cached of course, but fragmentation means you are likely to not find the entries in cache.

The other aspect that I haven't talked about is our fsync performance is kind of shit compared to other fs'es. Now this does get better in the nocow case but it's still pretty heavy and needs optimization.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#79

Post by ryao »

mrmondo wrote: post #66

- Very old kernel used (3.10!), makes me wonder how old the packages like btrfs-progs are as well. - BTRFS not mounted with compression (compress=lzo) - Don't use QCOW2, just don't, it's slow and you're just adding extra layers where you don't need to. It would be interesting to see you re-run these tests using a modern kernel, say at least 4.4 and either raw block devices or logical volumes along with mounting BTRFS…

The benchmark configuration appears to be designed to evaluate the use of storage technologies for a KVM host. Consequently, saying to use raw block devices when giving tips for improving btrfs performance is contradictory.

Also, there are a large number of people that will not run a newer kernel for several years because they are on RHEL6 or RHEL7, so while newer kernels are interesting, we should not discount the results on the basis that the kernel is old. The latest ZFSOnLinux code is able to run on those kernels, so while btrfs remains stagnant there, ZFS will continue to improve.

As for rerunning the tests, using recordsize=4K and compression=lz4 on ZFS should improve its performance here too. Putting the VM images on zvols (where it would be volblocksize=4K) rather than qcow2 also would help. In ZoL, zvols are block devices.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#80

Post by ryao »

ymse wrote: post #72
specialist wrote: post #45

Earlier quoted context omitted.

Is there something like Aphyr's Jespen for file systems? https://aphyr.com/tags/Jepsen I've been curious about ZFS, btrfs, etc. But as a layperson, I don't have the technical chops, gumption, wherewithal to figure what's what. Reading posts (comments) about the edge cases where they fail (data, performance, missing features) leaves me more baffled.

Not the same, but there was an article about file system fuzzing recently: https://news.ycombinator.com/item?id=11469535 The presentation is focused on how the fuzzing technique works rather than individual file system performance, but the "time to first bug" is quite telling. Copied here: ext4 (2h) XFS (1h45) GFS2 (8m) NTFS (4m) NILFS2 (1m) HFS (30s) HFS+ (25s) ReiserFS (25s) OCFS2 (15s) F2FS (10s) BTRFS (5s) I've h…

That sort of analysis is testing something on most of those that happen to be two different things on btrfs and ZFS. Specifically, corruption that passes checksums and corruption that does not pass checksums. btrfs' result was obtained by modifying it to disable checksums. How btrfs or ZFS do without the benefit of checksums should only matter for people exchanging disk images. In that case, it would be better to have a userland driver regardless of the filesystem. NetBSD is able to run its filesystem drivers in userspace for this reason.
Post reply on HN