Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

phoronix.com

Post reply on HN
Print view View original

141–150 of 231 posts

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#141

Post by boardwaalk »

As someone who is in the middle of pulling apart a BTRFS volume by hand (read: writing code to interpret the data structures) to try and recover it, I think being burnt enough is once.

No indication of any hardware issue: No recent power loss (& it's on a UPS), no SMART issues, no memory test positives. But the block tree (at least, WIP) is f*cked across all the disks (looks like two competing writers went at it) and none of the available tools can deal with it.

It wasn't a super exotic setup either: RAID10 with 4 disks (2 stripes), fairly full and regular snapshots/cleanup, but that's it.

I already converted my root to ext4 because paranoia and I'm probably going to move bulk data (what can be recovered) to ZFS.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#143

Post by mlyle »

dale_glass wrote: post #52
dekhn wrote: post #27

Earlier quoted context omitted.

Many folks I know who manage storage don't make the boot volume RAID (redundant)- instead, it's some rapidly duplicatable thing like an NMVE flash containing the root filesystem, and there's a replacement handy. Then you can bring up and bring the full power of userspace to bear on the RAID repair.

That seems like a very hacky way to accomplish something that should just work. I mean, it's better than nothing, but it seems far from ideal. If it's your own personal box, and you're okay with that fiddling, fine. But if people are supposed to get work done while the admin gets a new disk, it's not great.

There's pretty solid reasons why you might choose to have a very simple RAID1 set for a boot volume-- because you want to be sure to boot even if the huge array is degraded, and because you want faster I/O for the OS install, etc...

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#144

Post by encabulator »

dekhn wrote: post #27
dale_glass wrote: post #19

But is the RAID handling remotely sane yet? See https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... It has gems such as: * It won't boot on a degraded array by default, requiring manual action to mount it * It won't complain if one of the disks is stale * It won't resilver automatically if a disk is re-added to the array I think the first is the killer. RAID is a High Availability measure. Your system is…

Many folks I know who manage storage don't make the boot volume RAID (redundant)- instead, it's some rapidly duplicatable thing like an NMVE flash containing the root filesystem, and there's a replacement handy. Then you can bring up and bring the full power of userspace to bear on the RAID repair.

I just finished another late night "thanks" to unknown person who decided to use this approach several years ago while setting up 17 servers that I inherited. The /boot partition of these RHEL machines was put on SD cards, which have been gradually dying. So I got to boot the machine (which is 6000 km away) to recovery mode from ISO and re-create the /boot filesystem on HDD (RAID).

Of the 17 servers, only three are now remaining where I haven't had to do this. Not all of these were actual SD card failures, some where done preventively. Still, there have been several SD card failures requiring emergency repair work at inconvenient times. There have been 0 RAID failures requiring similar emergency work on the systems where /boot has been migrated to HDD based RAID.

Of course there is no replacement SD card handy with working /boot filesystem. Actually there is no such thing as "handy" in my case. I can actually reconstruct /boot partition on the HDD faster than anybody could go to the datacenter and replace the SD card. And if there were a replacement SD card, I would need to keep it up to date manually every time the kernel or initrd are updated.

I never want to see this kind of setup again, and frankly it felt insane the first time I saw it.

Of course I should have migrated all of the machines off the SD cards by now, but my excuse is that maintaining Linux on these machines is not really my responsibility (it is nobody's responsibility apparently, although many people are interested in keeping these systems online).

When I die I want to have a SD-card shaped tombstone.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#145

Post by coptun »

zeotroph wrote: post #108
coptun wrote: post #60

Earlier quoted context omitted.

I've been rocking raid 5/6 btrfs for years for my personal data, no problems apart from the speed. A scrub literally takes months.

Months? I thought it would just go through all the data and verify that the stored checksums match. Even if it fetches filedata more than once to ensure the parity bits are ok, how can it take longer than reading the entire disk a couple times?

Well this is just a desktop not a server, it isn't on 24/7.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#146

Post by justsomehnguy »

p1necone wrote: post #107
dale_glass wrote: post #45

Earlier quoted context omitted.

I don't quite understand the use case here. If I'm setting up RAID it's because I want the system to stay up. That's the only purpose for it. If a device goes missing for "unknown reasons", then the machine should still work, and I'll figure out what happened when monitoring pokes me and says RAID is degraded.

> That's the only purpose for it. That's not the only purpose for it. There's three reasons I can think of that you might set up a RAID array: * You want better uptime. (your use case) * You want to protect from data loss. (my assumption was that this is the most common use case, but I could be wrong. This also helps with uptime because there's nothing worse for uptime than having to restore lost data from a cold bac…

> Booting a RAID array with a failed disk is a bad idea

Booting a RAID array with a failed disk is absolutely fine idea.

How else I get access to the tools to identify the bad drive and resilver RAID on a replacement, be it in the same bay or not?

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#147

Post by RealStickman_ »

alerighi wrote: post #92
chomp wrote: post #47

Earlier quoted context omitted.

Why? Are you in a position where you can’t modify grub or fstab?

No, because I want to be elsewhere on a Sunday evening than in front of a computer modifying fstab to bring a production server back online.

Unless you want to reboot the server this shouldn't affect you in any way.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#148

Post by RealStickman_ »

dale_glass wrote: post #45
_kdave wrote: post #36

Earlier quoted context omitted.

> It won't boot on a degraded array by default, requiring manual action to mount it If you want it to behave like that then add 'degraded' to fstab. That a device is missing can have unknown reasons, the user should know better and resolve it or allow such boot. It's not automatic as there's no way to inform the user that it's degraded state.

I don't quite understand the use case here. If I'm setting up RAID it's because I want the system to stay up. That's the only purpose for it. If a device goes missing for "unknown reasons", then the machine should still work, and I'll figure out what happened when monitoring pokes me and says RAID is degraded.

The system stays up just fine. You just can't reboot it without fixing/ignoring the problem. I think that's fair.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#149

Post by cosmojg »

chasil wrote: post #55
DominoTree wrote: post #35

I've never lost data with btrfs, but I have purposefully avoided anything other than basic disk configurations. I use it on my workstations because it's the default in Fedora these days and haven't had any complaints. On the flip-side, I've done some of the most horrible things possible and screwed up my 30-disk ZFS array a number of times and I've never lost data. I doubt btrfs could recover from anything I've done…

Btrfs is able to do one important thing that ZFS cannot: defragment. I see XFS as the performance leader (appears on TPC.org the most often that I can see), btrfs as the fullest featured, and ZFS with the strongest reliability.

XFS is the best filesystem for people who don't want to deal with filesystems. Btrfs is there for the rest of us.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#150

Post by stingraycharles »

boardwaalk wrote: post #141

As someone who is in the middle of pulling apart a BTRFS volume by hand (read: writing code to interpret the data structures) to try and recover it, I think being burnt enough is once. No indication of any hardware issue: No recent power loss (& it's on a UPS), no SMART issues, no memory test positives. But the block tree (at least, WIP) is f*cked across all the disks (looks like two competing writers went at it) and…

For what it's worth, about a year ago I had data being corrupted with ZFS (something about NVMe + raidz + dedup if I recall correctly) -- this was fortunately while I was testing deployments, and quickly confirmed as a bug (and fixed) by the ZFS team. But it left me equally burned.

Using xfs|ext4 + mdraid is just a lot simpler and much faster, and it addresses most of my use cases.

Post reply on HN