Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux 5.10 BTRFS performance regression

reddit.com

Post reply on HN
Print view View original

171–180 of 202 posts

Re: Linux 5.10 BTRFS performance regression

#171

Post by bashinator »

nitrogen wrote: post #140
bashinator wrote: post #120

Earlier quoted context omitted.

Hardware RAID falls into the "never again" category for me.

Though in this case it sounds like any PCI card that couldn't handle the degraded signals from the angle connector would have exhibited data corruption. I have problems with USB packets getting corrupted on their way to my audio interface. Signal integrity is really underappreciated in the PC peripheral world.

I think /dev/md raid would have made that failure more noticeable, more quickly. Hardware RAID cards don’t always pass on disk errors to the kernel.

Re: Linux 5.10 BTRFS performance regression

#172

Post by BuildTheRobots »

xaduha wrote: post #76
BuildTheRobots wrote: post #70

Earlier quoted context omitted.

Where as I was always told to avoid hardware raid if possible, and that's been a mantra for 15+ years. The reason I was given is twofold. 1) If your RAID card dies it's nice to be able to plug drives into any controller and be able to access the data. 2) "look at how often MDADM fix bugs. Now look at how often your RAID card firmware gets fixed... use MDADM". Modern filesystems like btrfs & zfs both prefer to deal wi…

> Now look at how often your RAID card firmware gets fixed It was updated plenty, from 1.58(B) (earliest I can find) to 6.64(B) https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX-... I'm not recommending some abstract hardware RAID, I'm recommending this particular card for personal use, with backup obviously. Although if you just use RAID 0 or 1, then data is perfectly readable outside of RAID with a normal…

Even for home use I'm not sure I could recommend them. From experience using them in servers, I'd site a handful of problems:

+ Raid 5 requires BBWC + Raid 6 requires licensing + HPE requires a support contract to download most firmware + When used in servers, they report non-hp drives as being in a constant fault state.

Re: Linux 5.10 BTRFS performance regression

#173

Post by fomine3 »

zaarn wrote: post #86
kefyras wrote: post #80

Earlier quoted context omitted.

> but ZFS doesn't like living on a partitioned disk nearly as much. Don't know how true is this, since it's not even possible to create a zpool on the whole unpartitioned device on linux. It automatically creates GPT label with zfs and a small efi partition.

That is true, but I suspect that only happens to protect the zpool against tools like the Windows Repair Install (which likes to attempt repairs on disks without GPT label). Using ZFS on a partition inside a GPT disk tends to lead to a few quirky behaviours, like for example ZFS not always reporting correct SMART data of the device.

IIRC ZoL `zpool create /dev/disk/by-id/ata-XXX` makes gpt partition table and zfs partition inside it.

Re: Linux 5.10 BTRFS performance regression

#174

Post by magicalhippo »

StavrosK wrote: post #148
magicalhippo wrote: post #141

Earlier quoted context omitted.

That sucks, but also very weird. My pool, 2 vdevs each a 4-way RAID-Z1, has been used and abused for almost 7 years now. I've gone over the 85% mark but it still works fine, performance too. Some VM stuff but mostly media and similar. Do you mean slow IOPS or also sequential performance?

It's mostly sequential performance, but it's also very odd. Deleting a single 2 GB file sometimes takes tens of seconds, or copying files from one volume to another will again be extremely slow.. .

It does sound like fragmentation, which can still happen with a pool that's not filled under the "right" circumstances (usually related to workload vs configuration).

But yeah sadly that's the one area where ZFS is less stellar. Once it's fragmented it's hard to fix. Easiest is to send/receive to another pool, but as you note that is not always feasible.

Re: Linux 5.10 BTRFS performance regression

#175

Post by rleigh »

teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

I'd personally go with ZFS. There are already people in your field using ZFS to ensure the integrity of their data (based on past conversations here on HN).

I've used both ZFS and Btrfs extensively. Only Btrfs has lost data and behaved badly. I would highly recommend evaluating ZFS.

Re: Linux 5.10 BTRFS performance regression

#176

Post by rleigh »

Volundr wrote: post #101
thomaszander wrote: post #12

Earlier quoted context omitted.

> Why does btrfs have those issues compared to other filesystems? As someone that has built infrastructure on BtrFS for years, the scary stories are mostly just hot air and the stability of other filesystems is really not significantly better. Bugs like this happen, this is why Linus releases many release-candiates every kernel, this one got through as 10 was a rather massive kernel and there were several regressions…

In my experience, the stories are real. Our entire company was offline for a day when our central storage server quit accepting writes despite having over 50% free space. That's when I learned the hard way about the data/metadata split (something I was aware of but wasn't exactly top of mind) and BTRFS balance. You can certainly say it was my fault for not reading ALL the documentation before using BTRFS, and I'd fin…

I don't think it's your fault in any way.

We expect filesystems to work robustly. We do not expect them to fail after an arbitrary time interval merely by being used. Even terrible filesystems like FAT don't do that. They might get fragmented and slow, but they don't just stop. I find it incredible that this is often minimised by people; it's a complete show-stopper irrespective of the other problems Btrfs has.

I made exactly the same migration you did. ZFS has been solid, and it does exactly what it says on the tin.

Re: Linux 5.10 BTRFS performance regression

#177

Post by 08-15 »

teekert wrote: post #170
08-15 wrote: post #161

Earlier quoted context omitted.

Err, no. The CIGAR is not a checksum, even though you'll notice if it's corrupted, because it won't match the read length anymore. However, BAM is zlib-compressed (even if you asked samtools not to compress it). Each block therefore contains an actual checksum, and corruption will show up during decompression. Therefore, if you can decompress BAM, but the CIGAR doesn't make sense, you've got a software problem, not a…

Hmm, thank you for the information. But I'm by now pretty sure that I saw the checking process finish successfully at one moment, and not at others. It's not reproducible and all those algorithms should be deterministic, right? If not for bit flipping I really can't explain this.

It really can't be bit flipping on disk. First off, the checksum would catch it. Second, the file is compressed, so flipping a bit turns it into complete garbage. Third, BAM uses a length-prefixed coding in several places. A flipped bit in any length field would mess up the decoding.

Those bits could still flip in memory, which isn't BTRFS's fault. That's still unlikely, because you'd typically get segfaults, not silent corruption. So, it's probably bad software.

> those algorithms should be deterministic, right?

Combine bad code (samtools) with multithreading, and you will quickly become convinced that your computer is possessed by an evil spirit.

Re: Linux 5.10 BTRFS performance regression

#178

Post by rleigh »

throwii wrote: post #6
turminal wrote: post #4

Earlier quoted context omitted.

The problem with btrfs is that it has quirks like that (and worse, much worse) all the time.

Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/

> Why does btrfs have those issues compared to other filesystems?

Why? There are several reasons, but if you go right back to the beginning, there's a single reason which caused all the other problems: they started coding before they had finished the design.

All of the other problems are fallout from that. Changing the design and the implementation to fix bugs after the initial implementation was done. Introducing more bugs in the process. And leaving unresolved design flaws after freezing the on-disc format.

When you look at ZFS as a comparison, the design was done and validated before they started implementing it. Not unsurprisingly, it worked as designed once the implementation was done. Up-front design work is necessary for engineering complex systems, it really goes without saying.

This isn't even unique to Btrfs, but filesystems are one thing you can't hack around with without coming to grief; you have to get it right first time when their sole purpose is to store and retrieve data reliably. Many open source projects are ridden with problems because their developers were more interested in bashing out code than stopping and thinking beforehand. Same with a lot of closed source projects as well for that matter.

In the case of Btrfs, which was aiming from the start to be a "better ZFS", they didn't even take the time to fully understand some of the design choices and compromises made in ZFS, because they ended up making choices which had terrible implications. Examples: using B-trees rather than Merkle hashes; this is at the root of many of its performance problems. Not having immutable snapshots; again has performance implications as well as safety implications, and is rooted in not having pool transaction numbers and deadlists. Not separating datasets/subvols from the directory hierarchy; presents logistical and administration challenges, while ZFS datasets can freely inherit metadata from parents and the mount locations are a separate property. ZFS isn't perfect of course, there are improvements and new features that could be made, but what is there is well designed, well thought out, and is a joy to work with.

Re: Linux 5.10 BTRFS performance regression

#179

Post by rleigh »

kefyras wrote: post #80
zaarn wrote: post #46

Earlier quoted context omitted.

One thing Btrfs somewhat promised was mixed disk RAIDs. Ie, having a 2TB disk and a 1TB disk and being able to say "keep 2 copies of any data in this subvolume" and "keep only 1 copy in this subvolume". I'm currently on Bcachefs, which has erasure coding with similar promises and works better for me. ZFS is more rigid, I have to have all matched disks for best performance, plus RAM unless I like bad performance (I do…

> but ZFS doesn't like living on a partitioned disk nearly as much. Don't know how true is this, since it's not even possible to create a zpool on the whole unpartitioned device on linux. It automatically creates GPT label with zfs and a small efi partition.

It's nonsense. A ZFS pool is created from block devices, and ZFS doesn't care if those are whole discs or partitions.

Every ZFS pool I have is composed of GPT partitions, and this is also recommended in ZFS books as good practice to make it easier to identify a failed drive. Since you see the GPT partition name in the "zpool status" and other tools' output, it's handy.

Re: Linux 5.10 BTRFS performance regression

#180

Post by gregkh »

dlgeek wrote: post #104
gregkh wrote: post #71

Earlier quoted context omitted.

Where in the current CI that we have today is lacking that needs to be improved? We always want more testing and testers, what is preventing everyone from helping with this?

I'll bite: How can I help? I'm a software engineer who's not involved in Linux Kernel Dev... but I've got a stack of old laptops that I'd be happy to set up to run automated CI if that'd be helpful. Is there a webpage or doc somewhere I can look at? (I'm not trying to snark - the fact that you're you and you're here asking for help is making me want to dip my toe in).

Simplest thing to do, just run Linus's latest releases (the -rc releases), or from his git tree, on your machine and report any problem.

Second-simplest thing to do is to run the linux-next branch/tree on your machines and report any build warnings and runtime issues you find. That's what will be the "next" kernel releases and is where all of the developer/maintainer trees are merged together before they are sent to Linus.

Both of those should be very easy to do, and any problems found there should be easy to fix and resolve before they get to a "real" release.

Post reply on HN