I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (called cigar strings) and I am getting more and more convinced that the discrepancies I run into are actually bit flips on the hardware. Anybody else have any experience with this? BTRFS would be a good tool to prevent such bitrot, imo...
Linux 5.10 BTRFS performance regression
91–100 of 202 posts
Re: Linux 5.10 BTRFS performance regression
#92Earlier quoted context omitted.
Any rewrite must be carefully evaluated, true. And it may not be advisable. I'm trying to understand the issues. Are they not solvable because the kernel does not give enough guarantees as it gives to userspace, because the c-interfaces of the kernel have to be wrapped in unsafe or because of other reasons (architecture, data model, kernel constraints, ...)?
I think the issue is that you haven’t understood the issue, and just proposed using Rust. Don’t get me wrong, I’m a huge fan of Rust, but seeing people blindly suggest it as a panacea gets frustrating for people. The reality is that when people talk about Rust being fact, it’s because a decently optimised Rust program might be comparable to an equivalent C program. C is generally not the problem when it comes to perf…
> C is generally not the problem when it comes to performance issues in the kernel, which means rewriting that thing in Rust wouldn’t magically make it faster.
Agreed.
> The point being, it’s not clear yet, and to solve that requires understanding the problem, not effecting a needless rewrite in a new language.
Which is why the first question is why btrfs has issues others do not have. Some mention its CoW architecture, system design, too many features and not limiting storage to 90% and so increasing complexity. Others mention usual kernel issues.
Others mention that they had no issues to begin with and that its mixed reputation is unwarranted. I'm not clear who is right, but they are data points.
Thank you for your input in cautioning of rewrites to avoid needless work, I appreciate it.
Re: Linux 5.10 BTRFS performance regression
#93This version has got to be the worst kernel released in a while in terms of regression, from AMDGPU null pointer dereference crash[0] to f2fs data corruption bug[1] and now this. Fixes for these are on their way as far as I can tell but since the stable team are probably on Christmas vacation it might take a while. [0] https://bbs.archlinux.org/viewtopic.php?pid=1943906#p1943906 [1] https://bugzilla.kernel.org/show_b…
I get a vacation? Hah!
Thank you for your hard work Greg, it is very much appreciated.
Merry Christmas/Isaac Newton's Birthday!
Re: Linux 5.10 BTRFS performance regression
#94I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…
Depending on how you store this data (the number of physical disks involved) you may also want to explore ZFS. ZFS and BTRFS have pretty comparable features, but given how battle-hardened ZFS is, and its ever-increasing integration into the Linux ecosystem, I'm seeing less and less reason to prefer BTRFS.
Both do checksumming and regular scrubbing to help detect bitrot, but both need sufficient redundancy configured to actually be able to repair the corrupted data when it is detected.
Re: Linux 5.10 BTRFS performance regression
#95Earlier quoted context omitted.
> Is any software ever finished? Yes. When it does what it advertises, properly.
OpenZFS, in general, does what it advertises fairly well, and does not advertise features it does not possess. In what way has your experience differed?
My answer does not in any way reference the quality of ZFS. It only addressed the GP's question which I quoted.
Re: Linux 5.10 BTRFS performance regression
#96Earlier quoted context omitted.
Interesting decision they made, I wonder whether they would decide differently now after seeing all the complexity. So Rust does not decrease the complexity, but only removes certain kinds of errors which the compiler can detect. Neither logic errors nor speed regressions. Thank you for your input.
> I wonder whether they would decide differently now after seeing all the complexity. Decide what? Cow is a fundamental part of how btrfs works, and Rust didn't exist for the majority of Linux's life. (Although if you're into that, look at Redox)
Sorry I was unclear :) Maybe they would limit storage to 90% now, seeing that it increases complexity. Maybe I misunderstood the prior point though.
Thanks fr mentioning Redox!
Re: Linux 5.10 BTRFS performance regression
#97I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…
Or ZFS, I wouldn't store that kind of data on other filesystem's without that kind of protection...
Re: Linux 5.10 BTRFS performance regression
#98Earlier quoted context omitted.
I thought the Rust compiler solves issues that you wouldn't immediately see with pure C, which is why I had the idea. I didn't know this requires certain features which are not available inside the kernel. I only knew all existing interfaces may be unsafe because they are in C though. Rust does not seem as useful then. Thank you for your input.
It's not so much that rust the language requires them as much as it is that other non-rust parts can quite easily stomp all over the guarantees of rust without there ever being a way of knowing it happened. So rust alone won't solve many problems, but it would let you say "this code can't do these things itself", which is still a useful distinction. It also doesn't allow you to deal with misbehaving hardware that cha…
Do other parts stomp often? :) But true that can happen. Especially on non-ECC systems.
I didn't think about the hardware issues, hmm. I can't see how to do that, when the compiler guarantees get invalidated by hardware. Checks are also needed like in C? (assuming there are checks which do not get compiled out..)
Re: Linux 5.10 BTRFS performance regression
#99I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (c…
Bit-flips are a lot more common than people think as are read errors on drives. Surprisingly, usually a bit flipping here or there doesn't cause any real problems so they tend to go unnoticed. If your working with TBs of data using a checksumming filesystem like ZFS of BTRFS would certainly be recommended, but you should ALSO make sure your using ECC RAM. Checking the integrity of the data as it comes off the disk doesn't do you much good if a bit then flips while it's in memory.
Re: Linux 5.10 BTRFS performance regression
#100Assuming it can be fixed in a 5.10.x patch release, it's a rather minor story - a bug that is quickly found after release and fixed, hopefully. It will be investigated: https://marc.info/?l=linux-btrfs&m=160869337604422&w=2
The problem with btrfs is that it has quirks like that (and worse, much worse) all the time.
In 10 years of running a production service, the only two long duration outages were due to btrfs quotas (unwanted in the first place) slowing things to a crawl.