Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux 5.10 BTRFS performance regression

reddit.com

Post reply on HN
Print view View original

141–150 of 202 posts

Re: Linux 5.10 BTRFS performance regression

#141

Post by magicalhippo »

StavrosK wrote: post #110
magicalhippo wrote: post #40

Earlier quoted context omitted.

ZFS is great for storing data. It's very unlikely it will lose your data when using ZFS. It does however require a bit of care to maintain performance, and you need to know your expected workload going in. Otherwise you can find yourself in a situation with a very poorly performing pool where the only realistic route to recovery is a send/receive to a fresh pool and back. If you do not have a lot of sync workload (VM…

This is a home NAS with 50% free space, I rarely write stuff and don't even read much, yet it still managed to get super slow :( I don't use deduplication or compression, and the good people over at #zfs have failed to find anything wrong multiple times. I think recreating the entire thing is about my only option at the moment.

That sucks, but also very weird.

My pool, 2 vdevs each a 4-way RAID-Z1, has been used and abused for almost 7 years now. I've gone over the 85% mark but it still works fine, performance too.

Some VM stuff but mostly media and similar.

Do you mean slow IOPS or also sequential performance?

Re: Linux 5.10 BTRFS performance regression

#142

Post by ollran »

therealrootuser wrote: post #123
dmm wrote: post #54

Earlier quoted context omitted.

> - it takes 5-10 years for a filesystem or database to mature after it's released. Great news! BTRFS was introduced in the mainline kernel in March 2009! It's been ten years.

Agreed, Btrfs has been out for quite a while. I think it is only now beginning to become more widely accepted or noticed by the broader community, which is both a blessing and curse. That said, I'm going to continue to opt for Ext4 or XFS for the time being, even on distros where Btrfs is the default. Probably in a few years I'll get around to upgrading to Btrfs.

I use XFS too, but some large companies such as Facebook have adopted Btrfs on their servers.

https://lwn.net/Articles/824855/

Re: Linux 5.10 BTRFS performance regression

#143

Post by nitrogen »

gregkh wrote: post #71
est31 wrote: post #63

Earlier quoted context omitted.

On the bright side updating to 5.10 fixed a regression of a 5.4 to 5.8 kernel upgrade to me. The fix might have been in 5.9 but I only got the idea of upgrading after the 5.10 release. Anyways, Linux needs some more CI so that such bugs can be found during the RC phase.

Where in the current CI that we have today is lacking that needs to be improved? We always want more testing and testers, what is preventing everyone from helping with this?

I haven't been following kernel dev for years; what does the CI setup look like? Did the Phoronix Test Suite ever find its way into widespread use?

Back when I was building kernels for embedded hardware (Sheevaplug) in the 2.6.33 timeframe, I found a USB audio regression between 2.6.33.7 and later versions. If there were a semi-turnkey way to set up a testbench that could automatically reboot hardware in every new kernel, run through some basic tests, and report any deviation, I probably would have been more likely to do so. At the time I was working solo trying to release a polished consumer product (sadly though the product was released the business didn't work out) and didn't have time to dig into and report bugs.

Re: Linux 5.10 BTRFS performance regression

#144

Post by pstrateman »

teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

btrfs has eaten 100% of the data I have stored on it. 0/10 would not suggest.

ZFS has similar features and has eaten 0% of the data I have stored on it. would suggest.

Re: Linux 5.10 BTRFS performance regression

#145

Post by teekert »

ksk wrote: post #136
teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

What software do you use ? We use bwa/samtools/varscan/bcftools and we've reprocessed old data right from the fastq files a few times. So far we haven't gotten different results.

I encounter an error that I can't currently find exactly but it happens when running mismatch_profile from the RSeQC package. It reports a mismatch in the length of the cigar notation and the actual read inside a BAM file. Lately I have become 99% sure that this sometimes happens in files that did not show this error before (we reprocess a lot of data from time to time). I suspect it is bit flips because I think bit flips can easily go unnoticed, but not in this case, because the cigar notation is a shorter description of the actual read and they should thus match. The mismatch signals a change in the file that occurred without us touching it.

Re: Linux 5.10 BTRFS performance regression

#146

Post by tagrun »

panpanna wrote: post #13

With brtfs having all kind of crazy bugs and zfs never being finished, I guess it's back to ext4...

Back to ext4 because btrfs had crazy bugs?

ext4 had at least two critical data corruption bugs in the past 5 years in stable kernels (personal anecdote: one ruined the root filesystem on a server that I used, after which I stopped using ext4).

Re: Linux 5.10 BTRFS performance regression

#147

Post by fatboy93 »

gregkh wrote: post #56
esgwpl wrote: post #14

This version has got to be the worst kernel released in a while in terms of regression, from AMDGPU null pointer dereference crash[0] to f2fs data corruption bug[1] and now this. Fixes for these are on their way as far as I can tell but since the stable team are probably on Christmas vacation it might take a while. [0] https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 [1] https://bugzilla.kernel.org/show_b…

I get a vacation? Hah!

You should a small vacation sometime ;)

Bugs can always be fixed, mental health can't.

Thanks for all the awesome work Greg!

Re: Linux 5.10 BTRFS performance regression

#148

Post by StavrosK »

magicalhippo wrote: post #141
StavrosK wrote: post #110

Earlier quoted context omitted.

This is a home NAS with 50% free space, I rarely write stuff and don't even read much, yet it still managed to get super slow :( I don't use deduplication or compression, and the good people over at #zfs have failed to find anything wrong multiple times. I think recreating the entire thing is about my only option at the moment.

That sucks, but also very weird. My pool, 2 vdevs each a 4-way RAID-Z1, has been used and abused for almost 7 years now. I've gone over the 85% mark but it still works fine, performance too. Some VM stuff but mostly media and similar. Do you mean slow IOPS or also sequential performance?

It's mostly sequential performance, but it's also very odd. Deleting a single 2 GB file sometimes takes tens of seconds, or copying files from one volume to another will again be extremely slow.. .

Re: Linux 5.10 BTRFS performance regression

#149

Post by enz »

teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

As other folks are suggesting, ECC may be suitable for you (alongside a protective FS). But if you want to purchase your next workstation with ECC support, be very careful: some motherboards manufacturers just say 'ECC' in their specs but that doesn't mean that they actually use ECC capabilities, that only means that you can plug ECC-RAM into it. Make sure the specs explicitly say something like "Corrects single-bit errors and detects double-bit errors" (you need both, some MBs support only single-bit correction).

I think ECC RAM and a good FS are complementary tools to achieve storage high reliability. One without another may not be that useful.

Re: Linux 5.10 BTRFS performance regression

#150

Post by jpeloquin »

nicoburns wrote: post #31
jeromenerf wrote: post #20

Earlier quoted context omitted.

> Why does btrfs have those issues compared to other filesystems? Mostly because it has lots of features and as a consequence, is pretty large and complex. Closer to ZFS than ext2. Btrfs suffers from a initial bad rep, which is difficult to overcome.

ZFS itself seems very stable and not to suffer from these issues though.

Anecdotally, I've encountered issues with both ZFS and BTRFS at about the same rate. A public example of an apparent ZFS performance issue is https://github.com/openzfs/zfs/issues/9375 Both are much more quirky than simpler filesystems like ext4. Data integrity verification from checksumming makes it worth it though.

The ZFS vs. BTRFS choice, I think, depends more on whether you need specific features like offline deduplication or L2ARC / SLOG cache devices. And which one you're more familiar with (can troubleshoot better).

Post reply on HN