Linux 5.10 BTRFS performance regression • Hacker News Board

Re: Linux 5.10 BTRFS performance regression

#61

Post by ldrndll » Wed, Dec 23, 2020, 2:05 PM UTC

throwii wrote: post #38

turminal wrote: post #10

Earlier quoted context omitted.

I sincerely hope you are joking but I realize this mindset is quite common these days, so let me reiterate: Rust does not magically solve problems for you. Btrfs has a lot of issues and some of them may well be of the not possible in Rust sort, but I'm quite sure most of them are not and there is nothing Rust can fix about them. Rewriting a 13 years old and very complex thing in another language is a massive effort a…

Any rewrite must be carefully evaluated, true. And it may not be advisable. I'm trying to understand the issues. Are they not solvable because the kernel does not give enough guarantees as it gives to userspace, because the c-interfaces of the kernel have to be wrapped in unsafe or because of other reasons (architecture, data model, kernel constraints, ...)?

I think the issue is that you haven’t understood the issue, and just proposed using Rust. Don’t get me wrong, I’m a huge fan of Rust, but seeing people blindly suggest it as a panacea gets frustrating for people. The reality is that when people talk about Rust being fact, it’s because a decently optimised Rust program might be comparable to an equivalent C program. C is generally not the problem when it comes to performance issues in the kernel, which means rewriting that thing in Rust wouldn’t magically make it faster.

The issue will probably be some sort of pathological case in an algorithm being used, or perhaps from a poorly chosen algorithm. The point being, it’s not clear yet, and to solve that requires understanding the problem, not effecting a needless rewrite in a new language.

Re: Linux 5.10 BTRFS performance regression

#62

Post by mr_sturd » Wed, Dec 23, 2020, 2:07 PM UTC

gregkh wrote: post #56

esgwpl wrote: post #14

This version has got to be the worst kernel released in a while in terms of regression, from AMDGPU null pointer dereference crash[0] to f2fs data corruption bug[1] and now this. Fixes for these are on their way as far as I can tell but since the stable team are probably on Christmas vacation it might take a while. [0] https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 [1] https://bugzilla.kernel.org/show_b…

I get a vacation? Hah!

Merry Christmas, Greg!

Re: Linux 5.10 BTRFS performance regression

#63

Post by est31 » Wed, Dec 23, 2020, 2:18 PM UTC

esgwpl wrote: post #14

This version has got to be the worst kernel released in a while in terms of regression, from AMDGPU null pointer dereference crash[0] to f2fs data corruption bug[1] and now this. Fixes for these are on their way as far as I can tell but since the stable team are probably on Christmas vacation it might take a while. [0] https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 [1] https://bugzilla.kernel.org/show_b…

On the bright side updating to 5.10 fixed a regression of a 5.4 to 5.8 kernel upgrade to me. The fix might have been in 5.9 but I only got the idea of upgrading after the 5.10 release.

Anyways, Linux needs some more CI so that such bugs can be found during the RC phase.

Re: Linux 5.10 BTRFS performance regression

#64

Post by muxator » Wed, Dec 23, 2020, 2:28 PM UTC

gregkh wrote: post #56

esgwpl wrote: post #14

This version has got to be the worst kernel released in a while in terms of regression, from AMDGPU null pointer dereference crash[0] to f2fs data corruption bug[1] and now this. Fixes for these are on their way as far as I can tell but since the stable team are probably on Christmas vacation it might take a while. [0] https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 [1] https://bugzilla.kernel.org/show_b…

I get a vacation? Hah!

Thanks for your work, Greg!

Re: Linux 5.10 BTRFS performance regression

#65

Post by speakeron » Wed, Dec 23, 2020, 2:33 PM UTC

xaduha wrote: post #59

You can get 'surplus' HP P410 from Ali or Ebay, they are cheap and work just fine. Always was told that you can't beat hardware RAID and now that they are available people don't seem to be interested. Just looking at it I can say that apart from CPU/GPU chips themselves it's the best piece of electronics in my PC.

The problem with hardware RAID is that if the hardware fails, you need to replace that specific card with the same type to recover the data. If they're cheap enough and you want to go down that route, then buy two so you've got a backup.

On the server side, the tendency is now towards vertical striping (directly addressing disks on multiple servers through the network with software).

Re: Linux 5.10 BTRFS performance regression

#66

Post by circularfoyers » Wed, Dec 23, 2020, 2:41 PM UTC

thomaszander wrote: post #12

throwii wrote: post #6

Earlier quoted context omitted.

Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/

> Why does btrfs have those issues compared to other filesystems? As someone that has built infrastructure on BtrFS for years, the scary stories are mostly just hot air and the stability of other filesystems is really not significantly better. Bugs like this happen, this is why Linus releases many release-candiates every kernel, this one got through as 10 was a rather massive kernel and there were several regressions…

Does Fedora now have in-house expertise now that they've decided to make Btrfs default?

Re: Linux 5.10 BTRFS performance regression

#67

Post by curt15 » Wed, Dec 23, 2020, 2:49 PM UTC

circularfoyers wrote: post #66

thomaszander wrote: post #12

Earlier quoted context omitted.

> Why does btrfs have those issues compared to other filesystems? As someone that has built infrastructure on BtrFS for years, the scary stories are mostly just hot air and the stability of other filesystems is really not significantly better. Bugs like this happen, this is why Linus releases many release-candiates every kernel, this one got through as 10 was a rather massive kernel and there were several regressions…

Does Fedora now have in-house expertise now that they've decided to make Btrfs default?

Some Facebook btrfs devs agreed to help out. FB runs Fedora/CentOS + btrfs root on most of its systems.

Re: Linux 5.10 BTRFS performance regression

#68

Post by tremon » Wed, Dec 23, 2020, 2:50 PM UTC

diegocg wrote: post #60

knorker wrote: post #53

Earlier quoted context omitted.

I wouldn't count on that though. I'd say you were lucky. If the corruption happens before it gets sent to the disk, then you have checksummed and valid but broken data.

Btrfs has added some checks to prevent these issues. It can't catch everything, but some people have apparently hit them so it certainly helps https://btrfs.wiki.kernel.org/index.php/Tree-checker

The parent's point was that if the user data (not the tree metadata) was corrupted in-memory before writing to disk, no amount of checksumming will save you because the checksum was calculated over corrupted data.

Re: Linux 5.10 BTRFS performance regression

#69

Post by buran77 » Wed, Dec 23, 2020, 2:50 PM UTC

xaduha wrote: post #59

You can get 'surplus' HP P410 from Ali or Ebay, they are cheap and work just fine. Always was told that you can't beat hardware RAID and now that they are available people don't seem to be interested. Just looking at it I can say that apart from CPU/GPU chips themselves it's the best piece of electronics in my PC.

> Always was told that you can't beat hardware RAID

Recently with super powerful multi-core CPUs being the norm especially in places where RAID is a consideration, hardware RAID makes less and less sense. You just create additional complications like when moving your RAID array between 2 otherwise identical cards but with different firmwares fails. So you end up always buying spares and testing the migration before putting any data, then you freeze the config in place. On the other side you may be stuck forever with a buggy firmware.

Re: Linux 5.10 BTRFS performance regression

#70

Post by BuildTheRobots » Wed, Dec 23, 2020, 2:50 PM UTC

xaduha wrote: post #59

You can get 'surplus' HP P410 from Ali or Ebay, they are cheap and work just fine. Always was told that you can't beat hardware RAID and now that they are available people don't seem to be interested. Just looking at it I can say that apart from CPU/GPU chips themselves it's the best piece of electronics in my PC.

Where as I was always told to avoid hardware raid if possible, and that's been a mantra for 15+ years.

The reason I was given is twofold. 1) If your RAID card dies it's nice to be able to plug drives into any controller and be able to access the data. 2) "look at how often MDADM fix bugs. Now look at how often your RAID card firmware gets fixed... use MDADM".

Modern filesystems like btrfs & zfs both prefer to deal with the RAID aspects themselves as it gives them far more control. Even if you're a hardware raid afficianado, I wouldn't recommend formatting (eg) ZFS on top of a hardware raid pool.