Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux 5.10 BTRFS performance regression

reddit.com

Post reply on HN
Print view View original

181–190 of 202 posts

Re: Linux 5.10 BTRFS performance regression

#181

Post by gregkh »

nitrogen wrote: post #143
gregkh wrote: post #71

Earlier quoted context omitted.

Where in the current CI that we have today is lacking that needs to be improved? We always want more testing and testers, what is preventing everyone from helping with this?

I haven't been following kernel dev for years; what does the CI setup look like? Did the Phoronix Test Suite ever find its way into widespread use? Back when I was building kernels for embedded hardware (Sheevaplug) in the 2.6.33 timeframe, I found a USB audio regression between 2.6.33.7 and later versions. If there were a semi-turnkey way to set up a testbench that could automatically reboot hardware in every new ke…

We have so many different CI systems running on the kernel on a hourly basis.

We have the 0-day bot from Intel that runs so many things on all developer trees. We have kernelci running on many many different hardware platforms, and we have Linaro test systems also running on many different branches and hardware platforms.

If you want to tie your own hardware into the system, kernelci is the best place to start, I recommend looking into that.

thanks!

Re: Linux 5.10 BTRFS performance regression

#182

Post by xaduha »

BuildTheRobots wrote: post #172
xaduha wrote: post #76

Earlier quoted context omitted.

> Now look at how often your RAID card firmware gets fixed It was updated plenty, from 1.58(B) (earliest I can find) to 6.64(B) https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX-... I'm not recommending some abstract hardware RAID, I'm recommending this particular card for personal use, with backup obviously. Although if you just use RAID 0 or 1, then data is perfectly readable outside of RAID with a normal…

Even for home use I'm not sure I could recommend them. From experience using them in servers, I'd site a handful of problems: + Raid 5 requires BBWC + Raid 6 requires licensing + HPE requires a support contract to download most firmware + When used in servers, they report non-hp drives as being in a constant fault state.

> BBWC

why wouldn't you use it if you have it?

> Raid 6 requires licensing

Keys are out there and can be googled, in any case you probably don't need RAID 6 or any extra features for personal use

> HPE requires a support contract to download most firmware

Um, firmware for this card is on the page I linked, free as in beer

> When used in servers, they report non-hp drives as being in a constant fault state

I wouldn't know, haven't seen anything of the sort personally

Re: Linux 5.10 BTRFS performance regression

#183

Post by StavrosK »

magicalhippo wrote: post #174
StavrosK wrote: post #148

Earlier quoted context omitted.

It's mostly sequential performance, but it's also very odd. Deleting a single 2 GB file sometimes takes tens of seconds, or copying files from one volume to another will again be extremely slow.. .

It does sound like fragmentation, which can still happen with a pool that's not filled under the "right" circumstances (usually related to workload vs configuration). But yeah sadly that's the one area where ZFS is less stellar. Once it's fragmented it's hard to fix. Easiest is to send/receive to another pool, but as you note that is not always feasible.

Is there no defragmentation tool? I do have lots of free space.

Re: Linux 5.10 BTRFS performance regression

#184

Post by zaarn »

fomine3 wrote: post #173
zaarn wrote: post #86

Earlier quoted context omitted.

That is true, but I suspect that only happens to protect the zpool against tools like the Windows Repair Install (which likes to attempt repairs on disks without GPT label). Using ZFS on a partition inside a GPT disk tends to lead to a few quirky behaviours, like for example ZFS not always reporting correct SMART data of the device.

IIRC ZoL `zpool create /dev/disk/by-id/ata-XXX` makes gpt partition table and zfs partition inside it.

A stub table.

Re: Linux 5.10 BTRFS performance regression

#185

Post by teekert »

08-15 wrote: post #177
teekert wrote: post #170

Earlier quoted context omitted.

Hmm, thank you for the information. But I'm by now pretty sure that I saw the checking process finish successfully at one moment, and not at others. It's not reproducible and all those algorithms should be deterministic, right? If not for bit flipping I really can't explain this.

It really can't be bit flipping on disk. First off, the checksum would catch it. Second, the file is compressed, so flipping a bit turns it into complete garbage. Third, BAM uses a length-prefixed coding in several places. A flipped bit in any length field would mess up the decoding. Those bits could still flip in memory, which isn't BTRFS's fault. That's still unlikely, because you'd typically get segfaults, not sil…

:D

Currently not using btrfs by the way. But you are positive that BAM files should be self consistent? And when would you catch a bitflip then? What would it look like in a compressed file lake BAM? A much larger effect than just a cigar length mismatch supposedly?

Re: Linux 5.10 BTRFS performance regression

#186

Post by kasabali »

tagrun wrote: post #146
panpanna wrote: post #13

With brtfs having all kind of crazy bugs and zfs never being finished, I guess it's back to ext4...

Back to ext4 because btrfs had crazy bugs? ext4 had at least two critical data corruption bugs in the past 5 years in stable kernels (personal anecdote: one ruined the root filesystem on a server that I used, after which I stopped using ext4).

> ext4 had at least two critical data corruption bugs in the past 5 years in stable kernels

No it hadn't. Those bugs were lower in the stack in block layer.

Re: Linux 5.10 BTRFS performance regression

#187

Post by throwii »

rleigh wrote: post #178
throwii wrote: post #6

Earlier quoted context omitted.

Why does btrfs have those issues compared to other filesystems? I'd love to use btrfs too. Note that I deeply respect people who can write such complicated code, which I couldn't. Would Rust solve the non-speed issues? Rust-in-kernel discussion from August: https://lwn.net/Articles/829858/

> Why does btrfs have those issues compared to other filesystems? Why? There are several reasons, but if you go right back to the beginning, there's a single reason which caused all the other problems: they started coding before they had finished the design. All of the other problems are fallout from that. Changing the design and the implementation to fix bugs after the initial implementation was done. Introducing mo…

Can you tell how such evaluation on a design is done? Is some kind of formal verification, analysis or rather experimentation to figure out its properties normal?

Thank you for your input!

Re: Linux 5.10 BTRFS performance regression

#188

Post by simcop2387 »

throwii wrote: post #98
simcop2387 wrote: post #57

Earlier quoted context omitted.

It's not so much that rust the language requires them as much as it is that other non-rust parts can quite easily stomp all over the guarantees of rust without there ever being a way of knowing it happened. So rust alone won't solve many problems, but it would let you say "this code can't do these things itself", which is still a useful distinction. It also doesn't allow you to deal with misbehaving hardware that cha…

Do other parts stomp often? :) But true that can happen. Especially on non-ECC systems. I didn't think about the hardware issues, hmm. I can't see how to do that, when the compiler guarantees get invalidated by hardware. Checks are also needed like in C? (assuming there are checks which do not get compiled out..)

When the hardware can't make the guarantees, then software really can't do anything about it. There's really not any checks you can do, but modern hardware is getting the capabilities to try to prevent those kinds of issues with the IOMMU units, but operating system support is still hit or miss for most hardware and it won't prevent everything (just devices stomping on each-other with DMA). That's basically how the thunderbolt attacks have worked and the solutions to them.

Re: Linux 5.10 BTRFS performance regression

#189

Post by magicalhippo »

StavrosK wrote: post #183
magicalhippo wrote: post #174

Earlier quoted context omitted.

It does sound like fragmentation, which can still happen with a pool that's not filled under the "right" circumstances (usually related to workload vs configuration). But yeah sadly that's the one area where ZFS is less stellar. Once it's fragmented it's hard to fix. Easiest is to send/receive to another pool, but as you note that is not always feasible.

Is there no defragmentation tool? I do have lots of free space.

Sadly no, not really. Part of it is that it's complicated due to copy-on-write and snapshots. Other part is manpower, it has not been a high-priority issue for most production pools it seems. Google for "block pointer rewrite".

Assuming your free space fragmentation is not too bad (check output of zpool list, "frag" column is free space fragmentation level), you could just move files back and forth between the two. Assuming you don't have snapshots holding on to the files, this can help reduce the fragmentation.

Otherwise you're stuck with the send/receive.

Though I'd try the mailing list[1] to see if any of the gurus can help identify what's going wrong before attempting random ailments.

[1]: https://zfsonlinux.topicbox.com/groups/zfs-discuss/

Re: Linux 5.10 BTRFS performance regression

#190

Post by StavrosK »

magicalhippo wrote: post #189
StavrosK wrote: post #183

Earlier quoted context omitted.

Is there no defragmentation tool? I do have lots of free space.

Sadly no, not really. Part of it is that it's complicated due to copy-on-write and snapshots. Other part is manpower, it has not been a high-priority issue for most production pools it seems. Google for "block pointer rewrite". Assuming your free space fragmentation is not too bad (check output of zpool list, "frag" column is free space fragmentation level), you could just move files back and forth between the two. A…

Hmm, I'll try the mailing list, thank you for your help!
Post reply on HN