Live data from Hacker News

Linux 5.10 BTRFS performance regression

reddit.com

31–40 of 202 posts

Re: Linux 5.10 BTRFS performance regression

#31
post #6

Earlier quoted context omitted.

Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/

> Why does btrfs have those issues compared to other filesystems? Mostly because it has lots of features and as a consequence, is pretty large and complex. Closer to ZFS than ext2. Btrfs suffers from a initial bad rep, which is difficult to overcome.

ZFS itself seems very stable and not to suffer from these issues though.

Re: Linux 5.10 BTRFS performance regression

#32
post #4
post #2

Assuming it can be fixed in a 5.10.x patch release, it's a rather minor story - a bug that is quickly found after release and fixed, hopefully. It will be investigated: https://marc.info/?l=linux-btrfs&m=160869337604422&w=2

The problem with btrfs is that it has quirks like that (and worse, much worse) all the time.

I had been using btrfs for 5+ years without any issues.

Then a minor issue did happen, some files got missing after a system crash. That's not a problem in itself, that happens.

But then I discovered that the tooling of btrfs is awful to use. Tools to repair, restore, check, scrub, ... I think I got it to finally print some filenames of missing files after a huge amount of hoops. Just see how complex btrfs restore is: https://btrfs.wiki.kernel.org/index.php/Restore. Ideally those tools would know how btrfs works internally including all indirections and be able to translate that into recovering files to the user, not have, for example, a user manually puzzle together objectid numbers from a huge list.

But anyway, that made me simply format everything back to ext4 and go with that again for now.

So, I was a happy btrfs user until I discovered how bad its tooling is. No issues with bugs or anything though.

I really want btrfs to become better given that it's present in the kernel and has checksums, cow, .... I hope it'll keep improving!

Re: Linux 5.10 BTRFS performance regression

#33

Earlier quoted context omitted.

Is ZFS not ready for use? I was going to use it in production next year.

My home server is hardly production, but it has been running a ZFS raidz+1 for more than 5 years, and survived a disk failure. (Nb, avoid SMR hard drives - the rebuild took more than a week!)

Mine is RAIDZ too, and has become unusably slow since a few years back. Now I need to find a way to recreate the pool, but it's too much data to store on another disk... Not in love with ZFS so far.

Re: Linux 5.10 BTRFS performance regression

#34
post #5

The ratio of bad to good posts about BTRFS always made me wonder why anyone would roll with it. Don’t put your fs in crazy.

The ratio of bad to good posts has always stunned me too. But for the opposite reason. The linked post explains that there is a performance regression in a very rare and specific usecase: creating 100k files in a short span of time. And I'm not talking about unpacking boost, that's not enough files. Additionally, it will be fixed before any distro packages it. (even Arch is still on 5.9.14.) On the other hand btrfs g…

this is fair and i will cop to most of my exp in linux being in the "mainstream"/"commercial" filesystems. guilty as charged :)

Re: Linux 5.10 BTRFS performance regression

#35
post #6
post #4

Earlier quoted context omitted.

The problem with btrfs is that it has quirks like that (and worse, much worse) all the time.

Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/

My guess is this (compared to ZFS): With a CoW-file system like btrfs you have to problem that you need new file system space to delete something. This is problematic if the file system is full and you want to be able to write to it again by deleting something. ZFS solved this by just saying one can only fill a file system to 90% usage. At some point they even decreased this (during upgrade) and I had the issue that I couldn't write to the ZFS file system because this was lowered.

Btrfs tries to fully use the space and gets all the associated complexities. Additionally, because data/metadata ratio is not fixed one can get into situations where the file system is full and there is no more metadata space. For every action it needs to carefully check if there is enough space to actually perform the action even if the file system is nearly full. Improvements in this area caused this regression.

And no Rust wouldn't help. How often do you get a kernel Oops, dead lock or memory leak? Rust would help with those.

Re: Linux 5.10 BTRFS performance regression

#36

With brtfs having all kind of crazy bugs and zfs never being finished, I guess it's back to ext4...

Is any software ever finished? ZFS has been quite feature-full and production ready for more than a decade... why do you think it is not finished?

> Is any software ever finished?

Yes. When it does what it advertises, properly.

Re: Linux 5.10 BTRFS performance regression

#37
post #26

Earlier quoted context omitted.

Not btrfs and anecdotal but fedora 33 / 5.8 kernel / default gnome was unusable for me on amd r5 3500U. Headphone microphone didn’t work. Bluetooth ear phones and mic didn’t work. USB headphone microphone worked though. Sure enough most of the issues were resolved by 5.9 but I can no longer recommend Linux on the desktop.

With the problems you are listing, you would have never been able to recommend Linux. The problem stem from lack of driver support in the kernel for the devices on your machine. If that is the case, then your machine maker obviously does not support linux. So, it's no surprise that your machine had problems with linux. If you buy a machine that supports linux, you would not see those problems. Nothing to do with linu…

Sorry I failed to mention that things were much better before the update. Sure the fingerprint reader goodix never worked but the things I mentioned above are regressions. I’m all in favor of wayland but if we say we remove existing working features (fedora 32) because security or whatever we might as well wear tin foil hats and stay away from computers.

Re: Linux 5.10 BTRFS performance regression

#38
post #6

Earlier quoted context omitted.

Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/

I sincerely hope you are joking but I realize this mindset is quite common these days, so let me reiterate: Rust does not magically solve problems for you. Btrfs has a lot of issues and some of them may well be of the not possible in Rust sort, but I'm quite sure most of them are not and there is nothing Rust can fix about them. Rewriting a 13 years old and very complex thing in another language is a massive effort a…

Any rewrite must be carefully evaluated, true. And it may not be advisable. I'm trying to understand the issues.

Are they not solvable because the kernel does not give enough guarantees as it gives to userspace, because the c-interfaces of the kernel have to be wrapped in unsafe or because of other reasons (architecture, data model, kernel constraints, ...)?

Re: Linux 5.10 BTRFS performance regression

#39
post #6

Earlier quoted context omitted.

Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/

> Why does btrfs have those issues compared to other filesystems? As someone that has built infrastructure on BtrFS for years, the scary stories are mostly just hot air and the stability of other filesystems is really not significantly better. Bugs like this happen, this is why Linus releases many release-candiates every kernel, this one got through as 10 was a rather massive kernel and there were several regressions…

Thank you for posting your experience!

Re: Linux 5.10 BTRFS performance regression

#40

Earlier quoted context omitted.

My home server is hardly production, but it has been running a ZFS raidz+1 for more than 5 years, and survived a disk failure. (Nb, avoid SMR hard drives - the rebuild took more than a week!)

Mine is RAIDZ too, and has become unusably slow since a few years back. Now I need to find a way to recreate the pool, but it's too much data to store on another disk... Not in love with ZFS so far.

ZFS is great for storing data. It's very unlikely it will lose your data when using ZFS.

It does however require a bit of care to maintain performance, and you need to know your expected workload going in. Otherwise you can find yourself in a situation with a very poorly performing pool where the only realistic route to recovery is a send/receive to a fresh pool and back.

If you do not have a lot of sync workload (VMs, DBs) and don't have super-high performance needs, say a home NAS, then mainly you just need to think about not filling up the pool too much.

A nice way to do this is to create a root dataset where you set a quota to say 75% of capacity, and then create all other datasets below this one. You should not go above 85% space usage, as ZFS switches allocation strategy then to one which can significantly increase fragmentation.

If you do have a lot of sync writes, a SLOG device is basically mandatory. The SLOG device does not have to be large, it only stores about 5-10 seconds worth of writes, so 10-20GB can be plenty. I've partitioned up my SSDs and created a mirror out of two small partitions, using the remaining SSD space for other things.

One thing to keep in mind is that while an L2ARC device sounds like a great thing, depending on your configuration you can actually slow things down with one. A bunch of disks has a lot more bandwidth than a single SATA SSD. An L2ARC device also requires some memory overhead, so reduces your primary ARC. Again depending on load this can be detrimental.

And finally, don't ever think about using deduplication, unless you've read about the consequences, measured the performance benefits and ensured the memory overhead is acceptable. It sounds great on paper but has a lot of associated downsides that can ruin pool performance, and disabling it does not make it go away.

At least that's what I've picked up so far.

Post reply on HN