Linux 5.10 BTRFS performance regression • Hacker News Board

Re: Linux 5.10 BTRFS performance regression

#131

Post by Skunkleton » Wed, Dec 23, 2020, 6:26 PM UTC

gregkh wrote: post #89

tuldia wrote: post #81

Earlier quoted context omitted.

Every software has bugs, and is easier to criticize than to help. Don't focus on the negativity of HN and keep it up! Thanks for your hard work, Greg!

If people don't report bugs, we don't know they are there as it "works for me!". This isn't "negativity", this is people not understanding how the process works :) And you're welcome!

These are some attitude goals for me. It's so easy to take things personally. Being able to take things constructively even when they might be personal is a great skill.

Re: Linux 5.10 BTRFS performance regression

#132

Post by viraptor » Wed, Dec 23, 2020, 6:27 PM UTC

therealrootuser wrote: post #125

tremon wrote: post #74

Earlier quoted context omitted.

And it's been the default filesystem for openSUSE (system partitions only) since December 2015 [0], so there's been plenty of real-world usage too. [0] https://en.opensuse.org/Portal:13.2

Yeah, I remember that. I never do a separate home partition though, so I ended up with the entire disk in Btrfs. I wasn't paying as much attention as I should have when setting up partitions during installation, and didn't totally understand the implications of using Btrfs back then. Eventually though, it filled up my entire drive with snapshots, causing me much confusion and many disk full errors until I could figur…

The snapshots in this case come from suse, not btrfs itself. It's their package tooling creating them https://doc.opensuse.org/documentation/leap/reference/html/b...

If you're on lvm, you're still getting snapshots now, even with ext4.

Re: Linux 5.10 BTRFS performance regression

#133

Post by FGGTKingS » Wed, Dec 23, 2020, 6:31 PM UTC

thomaszander wrote: post #12

throwii wrote: post #6

Earlier quoted context omitted.

Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/

> Why does btrfs have those issues compared to other filesystems? As someone that has built infrastructure on BtrFS for years, the scary stories are mostly just hot air and the stability of other filesystems is really not significantly better. Bugs like this happen, this is why Linus releases many release-candiates every kernel, this one got through as 10 was a rather massive kernel and there were several regressions…

Yeah this is just fggtry from people who always keep their discs empty. Welcome to the real world lasshole, BTRFS is and always will be shit. Why choose it with ZFS and ext4 in the game? Might as well regress to Windows and their lucking Drive Services and NTFS and all of those skits. Choose a real drive my bigger. Merry christmas.

Re: Linux 5.10 BTRFS performance regression

#134

Post by viraptor » Wed, Dec 23, 2020, 6:33 PM UTC

curt15 wrote: post #11

encom wrote: post #8

Earlier quoted context omitted.

Nobody posts about things that are working fine.

Does OpenZFS have a similar ratio of bad to good posts?

It doesn't, but also their changelogs aren't seen by people who browse Linux changelogs. You'd look at https://github.com/openzfs/zfs/releases (which also contains a few bug fixes) if you're interested in zfs. Btrfs has quite a few more accidental readers.

Re: Linux 5.10 BTRFS performance regression

#135

Post by hda111 » Wed, Dec 23, 2020, 6:44 PM UTC

sethhochberg wrote: post #94

teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

Depending on how you store this data (the number of physical disks involved) you may also want to explore ZFS. ZFS and BTRFS have pretty comparable features, but given how battle-hardened ZFS is, and its ever-increasing integration into the Linux ecosystem, I'm seeing less and less reason to prefer BTRFS. Both do checksumming and regular scrubbing to help detect bitrot, but both need sufficient redundancy configured…

There is exactly one distro that comes with ZFS, so it's a bit exaggeration to say it's well integrated with Linux, considering how much effort is applied by kernel developers to make it even harder to use ZFS. And this one distro that includes ZFS is just one lawsuit away from removing it again.

Re: Linux 5.10 BTRFS performance regression

#136

Post by ksk » Wed, Dec 23, 2020, 6:51 PM UTC

teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

What software do you use ? We use bwa/samtools/varscan/bcftools and we've reprocessed old data right from the fastq files a few times. So far we haven't gotten different results.

Re: Linux 5.10 BTRFS performance regression

#137

Post by nitrogen » Wed, Dec 23, 2020, 6:58 PM UTC

teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

I have a lot of video files (e.g. old camera tapes) stored on spinning hard drives. Bit flips are distressingly common, and increase with age.

My next storage setup will definitely use a checksumming filesystem.

Re: Linux 5.10 BTRFS performance regression

#138

Post by fluffy87 » Wed, Dec 23, 2020, 6:58 PM UTC

esgwpl wrote: post #14

This version has got to be the worst kernel released in a while in terms of regression, from AMDGPU null pointer dereference crash[0] to f2fs data corruption bug[1] and now this. Fixes for these are on their way as far as I can tell but since the stable team are probably on Christmas vacation it might take a while. [0] https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 [1] https://bugzilla.kernel.org/show_b…

How is the kernel tested ? There weren’t any tests covering any of this ?

Re: Linux 5.10 BTRFS performance regression

#139

Post by jacob019 » Wed, Dec 23, 2020, 6:59 PM UTC

hda111 wrote: post #135

sethhochberg wrote: post #94

Earlier quoted context omitted.

Depending on how you store this data (the number of physical disks involved) you may also want to explore ZFS. ZFS and BTRFS have pretty comparable features, but given how battle-hardened ZFS is, and its ever-increasing integration into the Linux ecosystem, I'm seeing less and less reason to prefer BTRFS. Both do checksumming and regular scrubbing to help detect bitrot, but both need sufficient redundancy configured…

There is exactly one distro that comes with ZFS, so it's a bit exaggeration to say it's well integrated with Linux, considering how much effort is applied by kernel developers to make it even harder to use ZFS. And this one distro that includes ZFS is just one lawsuit away from removing it again.

It's easy to use zfs on Debian and derivatives

  apt install dkms spl-dkms
  apt install zfs-dkms zfsutils-linux

Re: Linux 5.10 BTRFS performance regression

#140

Post by nitrogen » Wed, Dec 23, 2020, 7:02 PM UTC

bashinator wrote: post #120

dsr_ wrote: post #105

Earlier quoted context omitted.

I once ran into a RAID card that would write bad data if you put it on a 90-degree PCI connector (needed for a 2U server box). No errors appeared anywhere until you encountered corrupted files.

Hardware RAID falls into the "never again" category for me.

Though in this case it sounds like any PCI card that couldn't handle the degraded signals from the angle connector would have exhibited data corruption.

I have problems with USB packets getting corrupted on their way to my audio interface. Signal integrity is really underappreciated in the PC peripheral world.