Live data from Hacker News

Five Years of Btrfs

markmcb.com

191–200 of 240 posts

Re: Five Years of Btrfs

#191
post #161

Earlier quoted context omitted.

I really don't understand the insane hype around ZFS. You can't read any thread that touches on filesystems without the ZFS zealots coming out.

Eh, i'm waiting for them to rewrite it in Rust.

This is such a meta-comment that I actually LOLd!

Re: Five Years of Btrfs

#192

Earlier quoted context omitted.

This was the killer feature for me. I had been wanting to try ZFS on my home NAS for a while (for snapshotting/redundancy/data integrity) and finally got enough disks that it made sense. I wasn't looking forward to learning what I presumed to be a very complicated system though. About 15 minutes into my research for setting up and maintaining a ZFS filesystem and I just went - wait thats it? So incredibly simple and…

ZFS is incredibly easy to learn to use, whereas btrfs is quite complicated to learn/use, and even more so if you've used ZFS since a lot of things are either just different enough to be weird, or so different that it makes no sense. Examples: ZFS snapshots can be recursive (-r) or not, whereas on btrfs they cannot be recursive; in discussions I've seen, this is mentioned as "a feature", since you can create a subvolu…

> but it also prevents you from dividing up a logical heirarchy into multiple behaviours (compression vs. not, block size, etc.).

Bind mounts can get around most of the limitations here, at the cost of polluting one directory with the canonical locations of all your special-purpose subvolumes. I think it's still awkward to simultaneously snapshot every subvolume that is mounted under a particular tree for incremental backup purposes.

Re: Five Years of Btrfs

#193
post #116

Earlier quoted context omitted.

It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). I'm not sure what causes it, but there seems to be an effect where btrfs loves you or hates you and few people with mixed experiences regarding data loss. One possible…

> It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). Why wouldn't you expect it to survive that? Is there a particular reason to believe those drives are broken? I.e., are they older consumer drives known to lie about c…

That's hyperbole, it can't be taken seriously. OpenSUSE uses Btrfs by default, if there were more problems outside what's expected by md+LVM+ext4 (or XFS), which is the feature comprised by Btrfs and then some, they wouldn't have made the on-going investments they have. Facebook has been using it in production with thousands of installations for years.

You want details from people experiencing zero problems, but you don't ask for details from people who are? That's a weird way to go about conducting the necessary autopsies, to discover and fix bugs.

Anyway, I monitor the upstream filesystems lists, and they all have bugs. They're all fixing bugs. They're all adding new features. And that introduces bugs that need fixing. It's not that remarkable, until of course someone suggests only one file system is to be avoided, while also providing no details, but depends on conjecture.

Re: Five Years of Btrfs

#194
post #116

Earlier quoted context omitted.

It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). I'm not sure what causes it, but there seems to be an effect where btrfs loves you or hates you and few people with mixed experiences regarding data loss. One possible…

In my experience, btrfs is very fragile in power loss or kernel crash/panic scenarios. It very consistently causes soft lockups on file read/writes after power loss until you run a `brtfs check --repair` on it. My experience is mostly on Arch, so it's not a case where it's out of date and missing patches.

Sounds like hardware problems in the storage stack. Btrfs developers contributed the dm-log-writes target to the kernel, expressly for conducting power loss tests on file systems. All the file systems benefit from this work. https://www.kernel.org/doc/Documentation/device-mapper/log-w... And Btrfs is doing the right thing these days.

I recently conducted system resource starvation tests where a compile process spun off enough threads to soak the system to the point it becomes unresponsive. I did over 100 forced power off tests while the Btrfs file system was being written to as part of that compile. Zero complaints: not on mount, not on scrubs, not with btrfs check, and not any while in normal operation following those power offs.

If you want to complain about Btrfs, complain about the man page warning to not use --repair without advice from a developer. You did know about that warning, right?

Re: Five Years of Btrfs

#195

Earlier quoted context omitted.

Not modifications per se, but it doesn't quite do the "normal" setup. Encryption is a mess (you can't export encrypted volumes via NFS), and the caching layer on top of it seems prone to corruption on the SSD (I've had my NVMe mirror cache drop twice over the last year and a half). I'd like to see them move to full disk encryption rather then their current approach.

They do encryption/compression on subvolume level; each share you create is a separate subvolume. For RAID5, they are using it on top of LVM, but with some modification - the synology implementation hooks LVM and btrfs together, so it gets ZFS-like properties. There's a guy on the internet, who was playing around with it: https://daltondur.st/syno_btrfs_1/

So they have fixed the last big hurdle to btrfs adoption in the small (single node) NAS space and are just sitting on it (violating the GPL). I urge any Synology user to write them to send you the Linux kernel source then upload it somewhere... though, their last Linux kernel drop seems to have been in 2017, so not much hope there...

Re: Five Years of Btrfs

#196

Earlier quoted context omitted.

Create a LVM thinpool, create a thin LV in that, format with XFS, put the database on top. Each time you want to a backup, create a CoW snapshot of the thin LV, then mount it somewhere and run the backup. The "main" thin LV should be happily chugging along independently when you are doing that. And all this is stable proven technology available about anywhere (eq. RHEL 7+).

My understanding is that performance on LVM drops dramatically after the first snapshot due to the way it handles CoW (synchronous writes on top of your async write). Is that no longer true, or only in certain circumstances? It seems as though the way to go would be to take a 'snapshot', back it up, and then delete it immediately; is that right? https://www.nikhef.nl/~dennisvd/lvmcrap.html

I think this complaint applies to the original LVM2 snapshots, not the new thin ones.

Re: Five Years of Btrfs

#197
post #183
post #116

Earlier quoted context omitted.

It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). I'm not sure what causes it, but there seems to be an effect where btrfs loves you or hates you and few people with mixed experiences regarding data loss. One possible…

>but there seems to be an effect where btrfs loves you or hates you Surely it depends on the btrfs implementation. e.g. Arch Linux getting daily kernel updates vs an enterprise distro

Just as unstable on Arch as of a month or two ago.

Re: Five Years of Btrfs

#198
post #95

I've had one issue with btrfs that took it off my radar completely. A customer had a runaway issue that filled a btrfs device with unimportant things. We found the errant process and killed it, but apparently if a btrfs device is completely full, you can't delete anything to free up space. File removal requires some amount of free space. Bricked the device, annoyed a customer, back to ext4.

A copy on write file system has this potential problem because nothing is overwritten. To delete anything requires space to write the metadata change reflecting the deletion, and before the data extends can be freed the change must be committed to stable media.

It's been years since Btrfs introduced "global reserve" which reserves enough metadata space to ensure it's possible to delete files on full file systems. But an old work around for this is to add a small device to the Btrfs volume, making is a 2 device volume. It could be a USB stick, a zram device (ramdisk), partition, or even a loop mounted file on some other file system. Delete the files, and now you can remove the temporary 2nd device.

Re: Five Years of Btrfs

#199
post #193

Earlier quoted context omitted.

> It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). Why wouldn't you expect it to survive that? Is there a particular reason to believe those drives are broken? I.e., are they older consumer drives known to lie about c…

That's hyperbole, it can't be taken seriously. OpenSUSE uses Btrfs by default, if there were more problems outside what's expected by md+LVM+ext4 (or XFS), which is the feature comprised by Btrfs and then some, they wouldn't have made the on-going investments they have. Facebook has been using it in production with thousands of installations for years. You want details from people experiencing zero problems, but you…

I asked RX14 why they called out their lack of problems as remarkable ("survive the most fucked up situations"). It sounds strange, as I mentioned.

I don't need to ask people who've had problems because I've had them myself, in unremarkable circumstances, a while back. I'm sure I could find reports on the mailing list as well, in which others have already asked for details.

Re: Five Years of Btrfs

#200
post #185

Earlier quoted context omitted.

One anecdote of a filesystem working fine and one anecdote of it becoming a disaster don't cancel each other out. I wouldn't buy a $5 USB thumb drive if half the people said it lost their data and half said it worked fine.

I'd buy it -but only for short-term use to sneakernet shit I already had backed up reliably somewhere else. of course, where we run into problems is that btrfs is meant to be the reliable backup. Oops.

You realize that there are $5 thumb drives that work, just like there are filesystems that actually work right? There isn't any benefit to using something broken, these problems have been solved.
Post reply on HN