Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

ilsistemista.net

Post reply on HN
Print view View original

31–40 of 85 posts

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#31

Post by ryao »

loeg wrote: post #22
josefbacik wrote: post #7

Earlier quoted context omitted.

Kvm (or any overwrite workload for that matter) is the worst possible workload for btrfs because of COW. We have ideas to address this but honestly it's not high on the list.

Shouldn't ZFS have similar problems with overwrite workloads?

> Shouldn't ZFS have similar problems with overwrite workloads?

ZFS does suffer from read-modify-write on partial record writes. The effect of that is apparent in the benchmarks. However, the benchmarks are being done on mechanical disks, which have low IOPS. The IOPS of a mechanical disk are roughly the same on a given sequence of IOs at different positions regardless of whether they are 4KB or 128KB in size, so it only has to pay a penalty once. If the record size were changed to 4KB, this penalty would disappear and ZFS performance should increase, provided that the VM internals are properly aligned.

Also, read-modify-write overhead reduces IOPS by at most 2 and bandwidth to the smaller of the link bandwidth and the IOPS times the record size. A CoW filesystem should be able to perform roughly at that level when it does read-modify-write on records/extents. Unless btrfs' internal extents are huge, there is an issue somewhere. Of course, having huge extents by default on which read-modify-write is done could also be considered a design issue.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#32

Post by ryao »

josefbacik wrote: post #20
SXX wrote: post #13

Earlier quoted context omitted.

This is sad since even with some performance loss compression and offline deduplication for VM images sounds like nice feature. Is there any docs that explain why COW make Btrfs so slow compared to QCOW2?

There are no docs. I'm not a qcow2 expert, what I know is very basic so anything I say about qcow2 can be very wrong. So qcow has a read only base image that gets updated when we change things. The image format just had the changes from the original image. So you update a package, it adds some metadata to point at the new stuff and adds the data in and you are done. So with btrfs you have this image on top of btrfs,…

That is a problem with CoW in general. ZFS uses a variable level indirect block tree. That puts an upper bound on fragmentation.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#33

Post by avar »

josefbacik wrote: post #12
biokoda wrote: post #9

Earlier quoted context omitted.

What kinds of workloads is it designed for then?

It's a general purpose file system, it does great with metadata heavy workloads and normal streaming writes. The overwrite case is special because of COW. Eventually we'll be comparable to everybody else but we aren't now.

I reinstalled my laptop with btrfs on / because "why not" about a year ago.

I ran into issues with maintaining a huge ~1M file Maildir, it seemed to do very badly with huge directories. Some kernel thread would be at 99% CPU while I was trying to populate a Maildir, and the entire system ground to a halt.

More importantly I would run into issues like running out of space on / with 20G left, but it was the "wrong kind of space". I.e. I had run out of metadata space but had plenty of file space left and had to run a rebalancing operation on the filesystem.

I didn't need any of the COW etc. advanced features that btrfs provides, and didn't test them, but as just a normal user needing a general purpose filesystem it was a bit too much hassle, especially with needing to administer the filesystem in a manner similar to how I might administer a RDBMS.

I ended up just going back to ext4.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#34

Post by babuskov »

_Codemonkeyism wrote: post #11
beefhash wrote: post #3

Why this article had to be segmented into 10 pages is beyond me, however.

The reason I didn't read the article.

Ditto. When I reach the end of page 1 on such website, I just close the tab and move on to something else.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#35

Post by kinkdr »

matt-attack wrote: post #5

What is with BTRFS? It stands out in every single test (and not in a good way).

Maybe it is not there yet, but given the momemtum it has, my gut feeling is that it will catch up eventually and be our default fs one day.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#36

Post by tadfisher »

avar wrote: post #33
josefbacik wrote: post #12

Earlier quoted context omitted.

It's a general purpose file system, it does great with metadata heavy workloads and normal streaming writes. The overwrite case is special because of COW. Eventually we'll be comparable to everybody else but we aren't now.

I reinstalled my laptop with btrfs on / because "why not" about a year ago. I ran into issues with maintaining a huge ~1M file Maildir, it seemed to do very badly with huge directories. Some kernel thread would be at 99% CPU while I was trying to populate a Maildir, and the entire system ground to a halt. More importantly I would run into issues like running out of space on / with 20G left, but it was the "wrong kind…

A subvolume for your Maildir, mounted with "-o nodatacow", would have prevented the first issue.

If btrfs could modify that behavior automatically using heuristics it would truly be general-purpose. However there are some inherent tradeoffs of that approach that users need to consider, and I consider that downside a basic constraint of the Unix filesystem abstraction.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#37

Post by kbenson »

kinkdr wrote: post #35
matt-attack wrote: post #5

What is with BTRFS? It stands out in every single test (and not in a good way).

Maybe it is not there yet, but given the momemtum it has, my gut feeling is that it will catch up eventually and be our default fs one day.

That sentiment is familiar, I used to subscribe to it when I was following this closely 3-4 years ago. That it hasn't changed in that time does not imbue hope.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#38

Post by tolle »

josefbacik wrote: post #23
loeg wrote: post #22

Earlier quoted context omitted.

Shouldn't ZFS have similar problems with overwrite workloads?

ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.

Wasn't ZFS even "production ready" before BTRFS development even began?

I don't remember the state of it when it was first introduced into Solaris, maybe someones memory is better then mine. Was ZFS better of in 06-07 Then BTRFS is now?

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#39

Post by ryao »

tolle wrote: post #38
josefbacik wrote: post #23

Earlier quoted context omitted.

ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.

Wasn't ZFS even "production ready" before BTRFS development even began? I don't remember the state of it when it was first introduced into Solaris, maybe someones memory is better then mine. Was ZFS better of in 06-07 Then BTRFS is now?

btrfs development started in 2007 while ZFS development started in 2001. If you want to do a point in time comparison, look at how ZFS was in 2010. You can get the last copy of OpenSolaris for the comparison. It would need to be done on physical hardware due to poor support for virtualization back then.

By the way, ZFS was deemed production ready after 4 years of development.

Re: ZFS, BTRFS, XFS, EXT4 and LVM with KVM – A Storage Performance Comparison (2015)

#40

Post by avar »

tadfisher wrote: post #36
avar wrote: post #33

Earlier quoted context omitted.

I reinstalled my laptop with btrfs on / because "why not" about a year ago. I ran into issues with maintaining a huge ~1M file Maildir, it seemed to do very badly with huge directories. Some kernel thread would be at 99% CPU while I was trying to populate a Maildir, and the entire system ground to a halt. More importantly I would run into issues like running out of space on / with 20G left, but it was the "wrong kind…

A subvolume for your Maildir, mounted with "-o nodatacow", would have prevented the first issue. If btrfs could modify that behavior automatically using heuristics it would truly be general-purpose. However there are some inherent tradeoffs of that approach that users need to consider, and I consider that downside a basic constraint of the Unix filesystem abstraction.

I think I did try stuff like that, including mounting / with nodatacow and other similar options.

I walked away with the impression that btrfs was a powerful tool for certain use-cases, but definitely not something I'd call a "general purpose" filesystem given the need to be fairly knowledgeable about its internals to use it for common desktop use-cases.

To me a general purpose filesystem is something like ext4, it doesn't have amazing performance, it's not bad either, but generally nothing unexpected happens with it and I can just leave it there and don't have to worry about it.

The btrfs filesystem seemed like the opposite of that. A very powerful tool whose power and flexibility made it less general purpose by virtue of needing to keep close tabs on how you were using it.

Post reply on HN