Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux 5.10 BTRFS performance regression

reddit.com

Post reply on HN
Print view View original

161–170 of 202 posts

Re: Linux 5.10 BTRFS performance regression

#161

Post by 08-15 »

teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

Err, no.

The CIGAR is not a checksum, even though you'll notice if it's corrupted, because it won't match the read length anymore. However, BAM is zlib-compressed (even if you asked samtools not to compress it). Each block therefore contains an actual checksum, and corruption will show up during decompression. Therefore, if you can decompress BAM, but the CIGAR doesn't make sense, you've got a software problem, not a hardware problem.

From my experience in beerinformatics, you got some software that... uhhh... interprets the BAM specification differently than you do. It's depressingly common.

Re: Linux 5.10 BTRFS performance regression

#162

Post by simion314 »

minot wrote: post #157
simion314 wrote: post #130

Earlier quoted context omitted.

You can't have it both ways, you can't get only the bug fixes without the new bugs , except on LTS you can if you know how experiment with getting a new kernel or video driver but not updating your DE,

> getting a new kernel > you can't get only the bug fixes without the new bugs > if you know how experiment with getting a new kernel or video driver I would agree with you if I was on my old machine with an nVidia graphics card which for what it is worth my new machine is a much better experience with AMD processor and integrated graphics but still how do you explain audio in failing to work as soon as I upgraded to…

What I would personally do is install Kubuntu LTS, then if I really need a new kernel because some hardware feature I enable a PPA that gives me that new kernel , if I need a different NVIDIA driver I install a PPA and try all the availleble drivers, find the one that works. Then if all works fine I don't upgrade the kernel or driver until I am forced and I always have the option to rollback,

Then when I want to try the next LTS years later I install it on a different partition and check if it works or not for my use case.

For my work I use Intellij , when there is a big update I get the .tag.gz and try it. If something goes wrong I still have the working version and I can go back.

Updating to latest and greatest was fun when I had the time and when I knew how to format the disk using "fdisk" and it was pleasurable to read and tweak stuff, this days I don't care about shiny stuff and my work does not require latest libraries and I am not using any first gen of hardware.

So IMO if you want stability and no surprises and the ability to maybe upgrade the kernel and a driver Ubuntu is a good solution(not sure about Debian or SUSE) and also I don't have experience with Wayland so maybe that invalidates things and is impossible to get a stable working Wayland setup.

Re: Linux 5.10 BTRFS performance regression

#163

Post by projct »

hda111 wrote: post #135
sethhochberg wrote: post #94

Earlier quoted context omitted.

Depending on how you store this data (the number of physical disks involved) you may also want to explore ZFS. ZFS and BTRFS have pretty comparable features, but given how battle-hardened ZFS is, and its ever-increasing integration into the Linux ecosystem, I'm seeing less and less reason to prefer BTRFS. Both do checksumming and regular scrubbing to help detect bitrot, but both need sufficient redundancy configured…

There is exactly one distro that comes with ZFS, so it's a bit exaggeration to say it's well integrated with Linux, considering how much effort is applied by kernel developers to make it even harder to use ZFS. And this one distro that includes ZFS is just one lawsuit away from removing it again.

Ubuntu, Proxmox, Manjaro all have root on ZFS via the installer and there are others that I don't remember offhand.

Re: Linux 5.10 BTRFS performance regression

#164

Post by malikolivier »

gregkh wrote: post #56
esgwpl wrote: post #14

This version has got to be the worst kernel released in a while in terms of regression, from AMDGPU null pointer dereference crash[0] to f2fs data corruption bug[1] and now this. Fixes for these are on their way as far as I can tell but since the stable team are probably on Christmas vacation it might take a while. [0] https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 [1] https://bugzilla.kernel.org/show_b…

I get a vacation? Hah!

Merry Christmas, Greg!

Thanks for all the hard work!

Re: Linux 5.10 BTRFS performance regression

#165

Post by projct »

zaarn wrote: post #46
flemhans wrote: post #16

Earlier quoted context omitted.

Is ZFS not ready for use? I was going to use it in production next year.

One thing Btrfs somewhat promised was mixed disk RAIDs. Ie, having a 2TB disk and a 1TB disk and being able to say "keep 2 copies of any data in this subvolume" and "keep only 1 copy in this subvolume". I'm currently on Bcachefs, which has erasure coding with similar promises and works better for me. ZFS is more rigid, I have to have all matched disks for best performance, plus RAM unless I like bad performance (I do…

On ZFS you can have multiple vdevs of different sizes in the same pool, and you can also set copies=N to have multiple copies of the data available for extra redundancy.

You can't unfortunately tell it that different disks have different speeds.

It also works just fine w/ a small (1GB or so) ARC if you want, just at an obvious performance penalty compared to having a larger cache.

Re: Linux 5.10 BTRFS performance regression

#166

Post by zaarn »

projct wrote: post #165
zaarn wrote: post #46

Earlier quoted context omitted.

One thing Btrfs somewhat promised was mixed disk RAIDs. Ie, having a 2TB disk and a 1TB disk and being able to say "keep 2 copies of any data in this subvolume" and "keep only 1 copy in this subvolume". I'm currently on Bcachefs, which has erasure coding with similar promises and works better for me. ZFS is more rigid, I have to have all matched disks for best performance, plus RAM unless I like bad performance (I do…

On ZFS you can have multiple vdevs of different sizes in the same pool, and you can also set copies=N to have multiple copies of the data available for extra redundancy. You can't unfortunately tell it that different disks have different speeds. It also works just fine w/ a small (1GB or so) ARC if you want, just at an obvious performance penalty compared to having a larger cache.

ZFS cannot have different sizes in a single vdev and I'm tired of having to explain this to people everytime I mention this issue.

Re: Linux 5.10 BTRFS performance regression

#167

Post by hda111 »

bluedino wrote: post #50
m1gu3l wrote: post #5

The ratio of bad to good posts about BTRFS always made me wonder why anyone would roll with it. Don’t put your fs in crazy.

Synology has been using BTRFS for a while, I haven't had any issues with it. https://www.synology.com/en-global/dsm/Btrfs

They use it just as a filesystem, like ext4. The RAID is still the classic Linux RAID because btrfs RAID isn't stable.

This means BTRFS isn't able to heal itself because there is no copy of the data that BTRFS could use.

I built my own NAS so I can just use ZFS with raidz, and when I ever have a silent data corruption the repair is done in a few seconds, not hours.

Re: Linux 5.10 BTRFS performance regression

#168

Post by rstuart4133 »

fluffy87 wrote: post #138
esgwpl wrote: post #14

This version has got to be the worst kernel released in a while in terms of regression, from AMDGPU null pointer dereference crash[0] to f2fs data corruption bug[1] and now this. Fixes for these are on their way as far as I can tell but since the stable team are probably on Christmas vacation it might take a while. [0] https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 [1] https://bugzilla.kernel.org/show_b…

How is the kernel tested ? There weren’t any tests covering any of this ?

> How is the kernel tested ? There weren’t any tests covering any of this ?

Despite appearances, "the kernel" is not a single monolithic thing. There is a about a 100 kloc core (but I haven't looked up that number in years). The rest, hardware drivers, network protocols, file systems, crypto, raid ... bolt on as modules.

Those modules are maintained separate teams. They are as related to the kernel as the phone dialler app is related to Android. The quality of each module is the responsibility of that team, not "the kernel" team. And that applies to testing the module as well.

In a sense, "the kernel" team is more like debian or redhat than developers. What they have done is develop a framework that lets them take bits created and maintained by a cast of thousands, and bolt it together into what appears to be a single coherent thing from the outside. So the answer to "how is the kernel tested" is "it's complex, and not centrally planned".

The other answer is what you are seeing is in fact part of the testing process. Most people use kernels packaged by their distribution. kernel.org releases are more like Microsoft's pre-releases of Windows. Most Debian users for example won't see it until it gets to Debian testing. To get there it must pass through Debian experimental (which is where 5.10 sits now) then sit in Debian unstable without bug reports for a while. Those release names should give you a hint about the anticipated stability of the kernel version. I personally won't use it until it takes another step, which is from Debian testing to Debian backports (which is when it because available to Debian stable users who are willing to risk compatibility issues).

This means that for for most users, 5.10 it's done yet as it has barely begun it's testing regime.

Re: Linux 5.10 BTRFS performance regression

#169

Post by teekert »

ksk wrote: post #156
teekert wrote: post #145

Earlier quoted context omitted.

I encounter an error that I can't currently find exactly but it happens when running mismatch_profile from the RSeQC package. It reports a mismatch in the length of the cigar notation and the actual read inside a BAM file. Lately I have become 99% sure that this sometimes happens in files that did not show this error before (we reprocess a lot of data from time to time). I suspect it is bit flips because I think bit…

Hmm, yeah that is odd! Maybe its a bug in the alignment tool?

It's not reproducible and I say I'm 99% sure it comes from bit flips is because I know with 99% certainty the process did finish successfully on a file at some point but then failed several weeks later, on the same file.

Re: Linux 5.10 BTRFS performance regression

#170

Post by teekert »

08-15 wrote: post #161
teekert wrote: post #91

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…

Err, no. The CIGAR is not a checksum, even though you'll notice if it's corrupted, because it won't match the read length anymore. However, BAM is zlib-compressed (even if you asked samtools not to compress it). Each block therefore contains an actual checksum, and corruption will show up during decompression. Therefore, if you can decompress BAM, but the CIGAR doesn't make sense, you've got a software problem, not a…

Hmm, thank you for the information. But I'm by now pretty sure that I saw the checking process finish successfully at one moment, and not at others. It's not reproducible and all those algorithms should be deterministic, right? If not for bit flipping I really can't explain this.
Post reply on HN