Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux 5.10 BTRFS performance regression

reddit.com

Post reply on HN
Print view View original

101–110 of 202 posts

Re: Linux 5.10 BTRFS performance regression

#101

Post by Volundr »

thomaszander wrote: post #12
throwii wrote: post #6

Earlier quoted context omitted.

Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/

> Why does btrfs have those issues compared to other filesystems? As someone that has built infrastructure on BtrFS for years, the scary stories are mostly just hot air and the stability of other filesystems is really not significantly better. Bugs like this happen, this is why Linus releases many release-candiates every kernel, this one got through as 10 was a rather massive kernel and there were several regressions…

In my experience, the stories are real. Our entire company was offline for a day when our central storage server quit accepting writes despite having over 50% free space. That's when I learned the hard way about the data/metadata split (something I was aware of but wasn't exactly top of mind) and BTRFS balance. You can certainly say it was my fault for not reading ALL the documentation before using BTRFS, and I'd find it hard to disagree, but any other filesystem wouldn't have had this problem.

I can't speak to if there are other foot-guns waiting around or how common problems like this are because we migrated back to FreeBSD and ZFS shortly after that experience. I do know they have since updated BTRFS to make that scenario less likely (but still not impossible).

Re: Linux 5.10 BTRFS performance regression

#102

Post by cat199 »

BuildTheRobots wrote: post #70
xaduha wrote: post #59

You can get 'surplus' HP P410 from Ali or Ebay, they are cheap and work just fine. Always was told that you can't beat hardware RAID and now that they are available people don't seem to be interested. Just looking at it I can say that apart from CPU/GPU chips themselves it's the best piece of electronics in my PC.

Where as I was always told to avoid hardware raid if possible, and that's been a mantra for 15+ years. The reason I was given is twofold. 1) If your RAID card dies it's nice to be able to plug drives into any controller and be able to access the data. 2) "look at how often MDADM fix bugs. Now look at how often your RAID card firmware gets fixed... use MDADM". Modern filesystems like btrfs & zfs both prefer to deal wi…

they are different animals .. 'right tool for the right job'

hardware raid has the hw complexity / pickyness you mention, but also is safer in crashes for checksumming raid levels (e.g. 4,5,6) since the stripe checksum happens from controller->disk in hardware rather than os->controller->disk in software, so you're less likely to lose a stripe due to OS issues (all else - like bugs - being equal)

also, IO wise, any mirrored raid is going to require N*mirrors of IO on the system bus, so you're more likely to saturate it

that said, with faster tech (NVMe), individual drives can easily saturate a single card, so its 'worth' paying the multiple IO pentalty multiplexed onto multiple individual lanes since the card is more likely to be the bottleneck than the bus

also, hw raid controllers even in jbod mode might be needed to get enough device fanout, though you're not using the controller for raid in that case

Re: Linux 5.10 BTRFS performance regression

#103

Post by oarsinsync »

Volundr wrote: post #99
teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

Bit-flips are a lot more common than people think as are read errors on drives. Surprisingly, usually a bit flipping here or there doesn't cause any real problems so they tend to go unnoticed. If your working with TBs of data using a checksumming filesystem like ZFS of BTRFS would certainly be recommended, but you should ALSO make sure your using ECC RAM. Checking the integrity of the data as it comes off the disk do…

To emphasise, using ECC RAM is another step that can be used to increase the integrity of your data.

It’s like security - it’s done in layers.

Using a checksumming filesystem without ECC RAM is an improvement on not using a checksumming filesystem at all. Do not consider ECC RAM to be a requirement for ZFS / BTRFS.

It does add another layer of improvement though, so if you can, you should!

And even if you can’t, and even if you can’t provide the redundancy to enable bitflips to be resolved, still use a checksumming filesystem if you can!

Re: Linux 5.10 BTRFS performance regression

#104

Post by dlgeek »

gregkh wrote: post #71
est31 wrote: post #63

Earlier quoted context omitted.

On the bright side updating to 5.10 fixed a regression of a 5.4 to 5.8 kernel upgrade to me. The fix might have been in 5.9 but I only got the idea of upgrading after the 5.10 release. Anyways, Linux needs some more CI so that such bugs can be found during the RC phase.

Where in the current CI that we have today is lacking that needs to be improved? We always want more testing and testers, what is preventing everyone from helping with this?

I'll bite: How can I help?

I'm a software engineer who's not involved in Linux Kernel Dev... but I've got a stack of old laptops that I'd be happy to set up to run automated CI if that'd be helpful.

Is there a webpage or doc somewhere I can look at?

(I'm not trying to snark - the fact that you're you and you're here asking for help is making me want to dip my toe in).

Re: Linux 5.10 BTRFS performance regression

#105

Post by dsr_ »

teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

I once ran into a RAID card that would write bad data if you put it on a 90-degree PCI connector (needed for a 2U server box). No errors appeared anywhere until you encountered corrupted files.

Re: Linux 5.10 BTRFS performance regression

#106

Post by vetinari »

xaduha wrote: post #59

You can get 'surplus' HP P410 from Ali or Ebay, they are cheap and work just fine. Always was told that you can't beat hardware RAID and now that they are available people don't seem to be interested. Just looking at it I can say that apart from CPU/GPU chips themselves it's the best piece of electronics in my PC.

Recently, I've switched P420i to HBA mode -- that's all ZFS running on that Proliant wanted.

Too bad it's not possible to boot from it in HBA mode too.

Re: Linux 5.10 BTRFS performance regression

#107

Post by the8472 »

teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

btrfs definitely helps catching errors, even without ECC or redundancy. An example: On a server with multiple VMs on a big consumer NVMe that was starting to get unreliable Windows reported nothing other than bluescreening occasionally (sfc and chkdsk showed no errors). Linux on the other hand reported errors during a bi-weekly scrub and in the kernel logs.

The SMART thresholds on the drive were strange too. A small but non-zero amount of read errors was not considered a health problem yet. I only realized that after the scrub reported problems.

Re: Linux 5.10 BTRFS performance regression

#108

Post by curt15 »

the8472 wrote: post #107
teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

btrfs definitely helps catching errors, even without ECC or redundancy. An example: On a server with multiple VMs on a big consumer NVMe that was starting to get unreliable Windows reported nothing other than bluescreening occasionally (sfc and chkdsk showed no errors). Linux on the other hand reported errors during a bi-weekly scrub and in the kernel logs. The SMART thresholds on the drive were strange too. A small…

>On a server with multiple VMs on a big consumer NVMe

How did you avoid the crippling performance penalty with storing VM disks on btrfs? The usual workaround suggested to btrfs users is to disable copy-on-write for VM images, but doing so also turns off checksumming thus disabling the very features that make btrfs, well, btrfs.

Re: Linux 5.10 BTRFS performance regression

#109

Post by snvzz »

yjftsjthsd-h wrote: post #85
snvzz wrote: post #21

HAMMER2 cannot be ready soon enough.

Is there any talk of porting to Linux?

No idea. I have only heard there is interest from openbsd.

At the time, Matt told them off because it's wasn't "ready for porting yet" and wanted to save them useless work.

Re: Linux 5.10 BTRFS performance regression

#110

Post by StavrosK »

magicalhippo wrote: post #40
StavrosK wrote: post #33

Earlier quoted context omitted.

Mine is RAIDZ too, and has become unusably slow since a few years back. Now I need to find a way to recreate the pool, but it's too much data to store on another disk... Not in love with ZFS so far.

ZFS is great for storing data. It's very unlikely it will lose your data when using ZFS. It does however require a bit of care to maintain performance, and you need to know your expected workload going in. Otherwise you can find yourself in a situation with a very poorly performing pool where the only realistic route to recovery is a send/receive to a fresh pool and back. If you do not have a lot of sync workload (VM…

This is a home NAS with 50% free space, I rarely write stuff and don't even read much, yet it still managed to get super slow :( I don't use deduplication or compression, and the good people over at #zfs have failed to find anything wrong multiple times.

I think recreating the entire thing is about my only option at the moment.

Post reply on HN