Live data from Hacker News

ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

ilsistemista.net

41–50 of 85 posts

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#41
post #22

Earlier quoted context omitted.

Shouldn't ZFS have similar problems with overwrite workloads?

ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.

The idea that btrfs has fewer contributors ought to be a surprise for many. Users tend to assume the opposite. ZFS having 5x more seems to be on the high side to me though. How did you determine that?

By the way, I was under the impression that ZFS development started in 2001 while btrfs development started in 2007. That would be a 6 year difference.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#42
Here a few things to note:

BTRFS has had Copy on Write disabled, while ZFS (not possible, cause the whole idea of the FS is intended to be copy on write), which actually makes BTRFS look even worse compared to ZFS, cause BTRFS writes once instead of twice and has most of its features not work, while still performance pretty bad. But then it's also younger.

Both ZFS and BTRFS (only know specifics of ZFS) can be configured to be better for DB workloads. On ZFS, I know a couple of people using it cause it has nice properties.

Anyway. You might not want to use ZFS or BTRFS for a (pure) database system when performance is the important thing (compared to data security).

What I find kind of missing is UFS because of the BSD world (it's kind of what Ext4 is in the sense of "your general purpose FS with good performance for DBs"), but okay.

And yeah, that all may sound a bit biased towards ZFS, but so far my experience with ZFS has been rather pleasent compared to BTRFS, but again. You might wanna use UFS or Ext4.

It's also kind of "wrong" to compare those file systems. It's like while you could use Redis and PostgreSQL for the same things it's probably not what you want to for one reason or the other.

But then of course it's good, cause you might really have a thing where you want to have a comparison of those two things to take the right decision for your specific application. Like those cases where you say "It's slow, but it makes a lot of thing easier" or "It doesn't guarantee that, but when I take care of it in the design I will only need only a fraction of the resources. And the other downsides don' t annoy me thaaat much".

Because the question came up. COW means Copy on Write and it really means what it says. You copy the data (so have find and to write blocks, duplicating your data on write) which in case of a full blown database which does that again (WAL, Autovacuum, keeping lots of metadata, etc.) you really shouldn't be surprised that it's a lot slower on write heavy sytems. It is more than expected.

On the other hand because the FS does a lot of things in a similar way to a database, other than snapshots and replication and all that cool stuff ZFS does things related to metadata extremely quickly and that's why some CDNs that have many small files use it. Besides just being an amazing thing for managing your data in general.

Also if you wanna learn more a about ZFS and have a good understanding of how you can run databases and other things on ZFS and really utilize it (not just gaining more performance) then I highly recommend the book FreeBSD Mastery: Advanced ZFS.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#43
post #8

Earlier quoted context omitted.

Not sure what you mean here. They showed that ZFS performed well on linux. Are you saying that ZFS performs poorly on BSD and Solaris? Love me some ZFS and it works well pretty much everywhere. OpenZFS shows good promise of keeping (bringing) the BSD, Illumos and Linux versions in line with each other.

> Are you saying that ZFS performs poorly on BSD and Solaris? I think LeoPanthera is saying it performs better on BSD (well FreeBSD) and Solaris (well Illumos-based in particular).

ClusterHQ's tests were unable to detect a difference between the platforms, with the exception of SELinux where performance was degraded unless xattr=sa was set. The code bases are almost identical, so performance differences tend to be caused by external factors and/or compatibility shims. There are also cases where one platform has a performance improvement before the others because the releases and porting are not done in lock step.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#44
post #33

Earlier quoted context omitted.

I reinstalled my laptop with btrfs on / because "why not" about a year ago. I ran into issues with maintaining a huge ~1M file Maildir, it seemed to do very badly with huge directories. Some kernel thread would be at 99% CPU while I was trying to populate a Maildir, and the entire system ground to a halt. More importantly I would run into issues like running out of space on / with 20G left, but it was the "wrong kind…

A subvolume for your Maildir, mounted with "-o nodatacow", would have prevented the first issue. If btrfs could modify that behavior automatically using heuristics it would truly be general-purpose. However there are some inherent tradeoffs of that approach that users need to consider, and I consider that downside a basic constraint of the Unix filesystem abstraction.

nodatacow means no checksums. Whatever performance problem it was intended to resolve should also recur the moment that a snapshot is done, because then CoW re-enters the picture.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#45
post #42

Here a few things to note: BTRFS has had Copy on Write disabled, while ZFS (not possible, cause the whole idea of the FS is intended to be copy on write), which actually makes BTRFS look even worse compared to ZFS, cause BTRFS writes once instead of twice and has most of its features not work, while still performance pretty bad. But then it's also younger. Both ZFS and BTRFS (only know specifics of ZFS) can be config…

Is there something like Aphyr's Jespen for file systems?

https://aphyr.com/tags/Jepsen

I've been curious about ZFS, btrfs, etc. But as a layperson, I don't have the technical chops, gumption, wherewithal to figure what's what. Reading posts (comments) about the edge cases where they fail (data, performance, missing features) leaves me more baffled.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#46

Earlier quoted context omitted.

But shouldn't NOCOW mitigate that? (which the article does for some tests)

NOCOW is horribly expensive because we still have to go check and make sure that there are no snapshots pointing at the changing extents. It only solves the fragmentation issue, and if you don't prealloc your image it doesn't even do that.

Are you sure that disabling CoW solves btrfs' fragmentation issue when you pre-allocate the image? If the volume is snapshotted regularly, CoW should be active once on each extent once until the next snapshot reactivates it. That should mean that the fragmentation would still occur, although more slowly. Is that correct or am I misunderstanding something?

I am only superficially familiar with btrfs internals, but I do not see any way to implement snapshots without either doing CoW or duplicating the data in its entirety. If you are checking to see if the data is part of a snapshot, then you should be doing CoW.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#47
post #40

Earlier quoted context omitted.

A subvolume for your Maildir, mounted with "-o nodatacow", would have prevented the first issue. If btrfs could modify that behavior automatically using heuristics it would truly be general-purpose. However there are some inherent tradeoffs of that approach that users need to consider, and I consider that downside a basic constraint of the Unix filesystem abstraction.

I think I did try stuff like that, including mounting / with nodatacow and other similar options. I walked away with the impression that btrfs was a powerful tool for certain use-cases, but definitely not something I'd call a "general purpose" filesystem given the need to be fairly knowledgeable about its internals to use it for common desktop use-cases. To me a general purpose filesystem is something like ext4, it d…

If you are open to trying again, I suggest trying ZFS. It should work well here.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#48
post #39
post #38

Earlier quoted context omitted.

Wasn't ZFS even "production ready" before BTRFS development even began? I don't remember the state of it when it was first introduced into Solaris, maybe someones memory is better then mine. Was ZFS better of in 06-07 Then BTRFS is now?

btrfs development started in 2007 while ZFS development started in 2001. If you want to do a point in time comparison, look at how ZFS was in 2010. You can get the last copy of OpenSolaris for the comparison. It would need to be done on physical hardware due to poor support for virtualization back then. By the way, ZFS was deemed production ready after 4 years of development.

That's true. One quick side note though: BTRFS actually had a fairly long "draft period" (multiple years) of design. I have no clue how long that was on ZFS. I just know that the original author mentioned that somewhere.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#49
post #8

Earlier quoted context omitted.

Not sure what you mean here. They showed that ZFS performed well on linux. Are you saying that ZFS performs poorly on BSD and Solaris? Love me some ZFS and it works well pretty much everywhere. OpenZFS shows good promise of keeping (bringing) the BSD, Illumos and Linux versions in line with each other.

> Are you saying that ZFS performs poorly on BSD and Solaris? I think LeoPanthera is saying it performs better on BSD (well FreeBSD) and Solaris (well Illumos-based in particular).

Another reason is that ZFS is tightly integrated with other things (for example jails and zones) on those systems, having switches related to other parts. I don't think that it is quiet there yet on Linux, even though I am sure that will come.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#50
post #48
post #39

Earlier quoted context omitted.

btrfs development started in 2007 while ZFS development started in 2001. If you want to do a point in time comparison, look at how ZFS was in 2010. You can get the last copy of OpenSolaris for the comparison. It would need to be done on physical hardware due to poor support for virtualization back then. By the way, ZFS was deemed production ready after 4 years of development.

That's true. One quick side note though: BTRFS actually had a fairly long "draft period" (multiple years) of design. I have no clue how long that was on ZFS. I just know that the original author mentioned that somewhere.

If you have time, you can watch this video where Jeff Bonwick describes the birth of ZFS:

http://m.youtube.com/watch?v=dcV2PaMTAJ4

I am the guy who asked the LZJB question. To summarize my recollection, formal design work on ZFS started in 2001 when Matthew Ahrens started working at Sun. Jeff Bonwick had promised Matthew Ahrens a job at Sun making a filesystem a 6 months to a year before then when Matt was still in college. I am sure that both Jeff and Matt had some thoughts on it during that time, but there was no formal effort until Matt's employment started.

Post reply on HN