Live data from Hacker News

Five Years of Btrfs

markmcb.com

221–230 of 240 posts

Re: Five Years of Btrfs

#221
post #207
post #111

Earlier quoted context omitted.

The hype is quite easy to understand. Snapshots and checksums are two complete game-changers. ZFS has them both. And there are no real alternatives in many cases. I've personally waited for BTRFS longer than a decade but my use-cases are yet to be considered stable (not something you really mess with in regard to filesystems). Honestly, as sure as I have been on the success of BTRFS I now consider BTRFS dead on arriv…

What warts do you speak of?

No defragmentation, and as far as I'm aware all copy-on-write filesystems suffer greatly from fragmentation once utilization goes too high. ZFS will never recover unless you restart from scratch.

No way to rebalance a pool. Also increasing a pool always results in less reliability (in terms of drive losses that results in the whole pool going down).

No proper recovery tools if something goes wrong.

Then the lack of flexibility talked about in the article. This means the up-front cost and total cost is vastly more than a more typical setup where you can buy drives spread out over many years and take advantage of falling prices, less power consumption and noise (in part because you typically start such an array with higher density drives, since the low cost and longevity allows you to).

Probably forgot some other reasons.

That said I still use zfs (freenas) at home. But because of the above it is quite hard to blindly recommend it.

Re: Five Years of Btrfs

#222
post #216

Earlier quoted context omitted.

In my experience, btrfs is very fragile in power loss or kernel crash/panic scenarios. It very consistently causes soft lockups on file read/writes after power loss until you run a `brtfs check --repair` on it. My experience is mostly on Arch, so it's not a case where it's out of date and missing patches.

Since a year I literally daily turn of my machine by pulling a plug (home automation turns off all plugs at midnight to make me go bed ;). My quite large 1tb multivolume, multisnapshot BTRFS fs never had any problems. And it's quite aggressive cfg (big fs commit). P.S. I do have backups though.

Ugh. You are testing your home the Netflix way [1] :-)

Why not putting poweroff in a cron task a bit before midnight so you don't uselessly risk hosing your file system? You can always restore your backup but it takes time!

[1] https://arstechnica.com/information-technology/2012/07/netfl...

Re: Five Years of Btrfs

#223
post #90
post #65

Earlier quoted context omitted.

> When Sun released ZFS as open source, they made a deliberate decision to use a license that prevented it from being integrated into the Linux kernel This is simply totally false no matter how many times people repeat it. It pure FUD. Sun picked the licence because they had to allow linking with closed code for their products, going with the GPL was simply not viable given the situation with drivers on their platfor…

Sun's motivation for choosing the CDDL is beside the point. Unless ZFS is released under a license that allows it to be redistributed under the GPL, ZFS cannot be legally built into Linux as a filesystem. If you have reason to believe that Linux developers can go ahead and simply integrate ZFS into Linux without worrying about the license, I'm sure lawyers from the FSF, IBM, Cannonical, etc would love to hear your ex…

This is not a new thought and the lawyers actually understand this. GPL was not designed to protect user from open source. And its an idiotic missapplication. Oracle themself diliver ZFS with Linux, and so do many, many others.

The only 'argument' is that we can't do it because 'big bad oracle' will sue you but that really doesn't hold up.

Re: Five Years of Btrfs

#224
post #50
post #32

Earlier quoted context omitted.

You can grow and add/remove raid1 devices (mirror vdevs) in ZFS without any significant work or downtime. Shrinking does require a bit more work, but depending on your setup it can be done fairly painlessly with send/recv (and shrinking is usually not something which is a very common administrative operation).

"fairly painlessly" and "without significant work or downtime" doesn't sound like it lines up with btrfs's, which I would describe as "one command and zero downtime (just some io load if you rebalance immediately)" for both operations. btrfs is also mainline, which increases how painless it is to use. BTRFS does have some scary stories from earlier in its development, and true raid5 seems like it's unlikely to be saf…

The first operations I mentioned (adding or removing a device from a vdev, or adding a new vdev) are one command with no downtime:

  % zpool attach    // add to a vdev
  % zpool detach    // remove from a vdev
  % zpool add    // add a new vdev
In the newest ZFS versions, you can also remove mirror and singleton vdevs (this does require some time -- because the data needs to be copied from the drives) but it's all done in the background:

  % zpool remove  
Shrinking a pool "the old way" (which is still sometimes necessary depending on what you're doing) is definitely more involved -- you have to create a new pool with the layout you need and then do a zfs send/recv from your old pool to the new one. This does only take a handful of commands but I would definitely consider it to be a much more complicated affair than the operations I mentioned above.

I would not (nor did I) compare LVM (or md-raid) to btrfs or ZFS -- those technologies have fundamental limitations regarding the integrity of your data that ZFS (and btrfs) don't have. And don't get me wrong -- I don't have a problem with btrfs (I run btrfs on all of my machines except my home server -- which runs ZFS), I just disagree with GP's point that ease of use is an argument for btrfs over ZFS. There are many arguments for either technology.

> btrfs is also mainline, which increases how painless it is to use.

I agree that this is one argument to pick btrfs over ZFS (though on most distributions it isn't really that hard to install ZFS, the fact that btrfs requires zero extra work to use on Linux is a benefit).

Re: Five Years of Btrfs

#225
post #153
post #97

Earlier quoted context omitted.

ZFS had this issue (I believe fixed) workaround was to pick up one large file that you wanted to delete and do `echo -n > /the/unimportant/file` once the file was reduced in size to 0, rm started to work again. Not sure if that workaround would work in btrfs, but it worked on ZFS.

ZFS reserves 1/64 of every disk precisely so it can't be truly fully allocated. It leaves enough room to delete snapshots, truncate files, and so forth. Mind that everything is copy-on-write, you can't do anything, even metadata changes, without allocating new blocks. It needs the reserve space.

I had a ZFS bug once where they increased the amount reserved in a new release which caused my file system to be 100% and me unable to delete anything until I went back to the previous release.

Btrfs uses the the disk completely. This is harder to do (also compared to e.g. ext4 reserving a fixed amount of inode space which may be unused when the disk is full). At some point they added an in-memory "global reserve" metadata space which allows you to delete stuff even if the file system is full.

Re: Five Years of Btrfs

#226
post #32

Earlier quoted context omitted.

You can grow and add/remove raid1 devices (mirror vdevs) in ZFS without any significant work or downtime. Shrinking does require a bit more work, but depending on your setup it can be done fairly painlessly with send/recv (and shrinking is usually not something which is a very common administrative operation).

How? My understanding is that you create a new vdev and add the old vdev as a device, basically recursively creating volumes with each new device you add.

[deleted]

Re: Five Years of Btrfs

#227
post #32

Earlier quoted context omitted.

You can grow and add/remove raid1 devices (mirror vdevs) in ZFS without any significant work or downtime. Shrinking does require a bit more work, but depending on your setup it can be done fairly painlessly with send/recv (and shrinking is usually not something which is a very common administrative operation).

How? My understanding is that you create a new vdev and add the old vdev as a device, basically recursively creating volumes with each new device you add.

Which operation are you asking about? [1] is a sister comment which I posted that outlines how to do most of the operations I mentioned.

[1]: https://news.ycombinator.com/item?id=22168494

Re: Five Years of Btrfs

#228
This article makes a few mistakes with regards to ZFS. Some are understandable (the author presumably last looked at the state of ZFS 5 years ago), but some were not true even 5 years ago:

> If you want to grow the pool, you basically have two recommended options: add a new identical vdev, or replace both devices in the existing vdev with higher capacity devices.

You can add vdevs to a pool which are different types or have different parities. It's not really recommended because it means that you're making it harder to know how many failures your pool can survive, but it's definitely something you can do -- and it's just as easy as adding any other vdev to your pool:

  % zpool add   
This has always been possible with ZFS, as far as I'm aware.

> So let’s say you had no writes for a month and continual reads. Those two new disks would go 100% unused. Only when you started writing data would they start to see utilization

This part is accurate...

> and only for the newly written files.

... but this part is not. Modifying an existing file will almost certainly result in data being copied to the newer vdev -- because ZFS will send more writes to drives that are less utilised (and if most of the data is on the older vdevs, then most reads are to the older vdevs, and thus the newer vdevs get more writes).

> It’s likely that for the life of that pool, you’d always have a heavier load on your oldest vdevs. Not the end of the world, but it definitely kills some performance advantages of striping data.

This is also half-true -- it's definitely not ideal that ZFS doesn't have a defrag feature, but the above-mentioned characteristic means that eventually your pool will not be so unbalanced.

> Want to break a pool into smaller pools? Can’t do it. So let’s say you built your 2x8 + 2x8 pool. Then a few years from now 40 TB disks are available and you want to go back to a simple two disk mirror. There’s no way to shrink to just 2x40.

This is now possible. ZoL 0.8 and later support top-level mirror vdev removal.

> Got a 4-disk raidz2 pool and want to add a disk? Can’t do it.

It is true that this is not possible at the moment, but in the interest of fairness I'd like to mention that it is currently being worked on[1].

> For most fundamental changes, the answer is simple: start over. To be fair, that’s not always a terrible idea, but it does require some maintenance down time.

This is true, but I believe that the author makes it sound much harder than it actually is (it does have some maintenance downtime, but because you can snapshot the filesystem the downtime can be as little as a minute):

    # Assuming you've already created the new pool $new_pool.
    % zfs snapshot -r $old_pool/ROOT@base_snapshot
    % zfs send $old_pool/ROOT@base_snapshot | zfs recv $new_pool/ROOT

    # The base copy is done -- no downtime. Now we take some downtime by stopping all use of the pool.
    % take_offline $old_pool # or do whatever it takes for your particular system
    % zfs mount -o ro $old_pool/ROOT # optional
    % zfs snapshot -r $old_pool/ROOT@last_snapshot
    % zfs send -i @base_snapshot $old_pool/ROOT@last_snapshot | zfs recv $new_pool/ROOT

    # Finally, get rid of the old pool and add our new pool.
    % zpool export $old_pool
    % zpool import $new_pool $old_pool
    % zfs mount -a # probably optional
[1]: https://www.youtube.com/watch?v=Njt82e_3qVo

Re: Five Years of Btrfs

#229

Earlier quoted context omitted.

This. In fact, it's really common to use a ZFS array on single nodes, and then create a SAN using multiple such machines by layering Ceph on top.

That's interesting, but it's layers upon layers... (RIP latency), I think. Unless it's about just bandwidth and volume, then latency is not that big of a deal.

You don't have to use ZFS snapshots. I haven't run a system like this in production but presumably you choose ZFS because it's flexible in how you configure the arrays (as is say, LVM) and because it supports checksumming.

Re: Five Years of Btrfs

#230

I went on a quest a few years ago, thinking it would be good for the industry to standardize on a single next generation filesystem for UNIX. I started with ZFS on linux since that seemed to have the most vocal advocates. That lasted about a half year, until a bug in the code resulted in a completely corrupt disk, and I had to restore 4TB of data over a month from offside backups. That plus the licensing confusion ar…

I really don't understand the insane hype around ZFS. You can't read any thread that touches on filesystems without the ZFS zealots coming out.

I had to check the dictionary for the meaning of "hype"

a situation in which something is advertised and discussed in newspapers, on television, etc. a lot in order to attract everyone's interest:

May be its just me because Morden day usage of "hype" seems to involve and implies a negative meaning, especially in tech. Similar to false advertising. And no one was actively promoting ZFS, they were only very "responsive".

And then zealots, I had to reread 226 comments, ran to Cambridge dictionary

a person who has very strong opinions about something, and tries to make other people have them too

I dont see anyone having strong opinions and force others to have the same. If anything a lot of people are showing not because the love ZFS, but they have been burnt by btrfs.

Post reply on HN