Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability • Hacker News Board

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#211

Post by awiesenhofer » Tue, Dec 13, 2022, 6:03 PM UTC

justsomehnguy wrote: post #151

cycomanic wrote: post #118

Earlier quoted context omitted.

Hold on, people make this out to be like if a disk fails it requires immediate intervention. However if the system already booted it stays up. Who reboots a production system without monitoring it? Also if you boot with a degraded disk are you not asking for massive trouble, because if another disk fails you might end up with data loss which is IMO much worse on a production system than not being able to boot until y…

> Who reboots a production system without monitoring it? Anyone. > Also if you boot with a degraded disk are you not asking for massive trouble > not being able to boot until you add another disk Great. I'm just a [sys]admin who were given a task to do something on $server. I jump around the red tape, claw out a 15 minute downtime because the task requires reboot, proceed with all that corporate dance with notificati…

As someone who too had all of these happen at one point or another over the years: These are all process issues, not technical issues. No matter your RAID config or FS features,you're gonna have a bad time.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#212

Post by eikenberry » Tue, Dec 13, 2022, 7:47 PM UTC

mekster wrote: post #95

eikenberry wrote: post #89

Earlier quoted context omitted.

Ideally on a ZFS replacement that has a compatible license.

Using Ubuntu is hell of a lot faster than waiting for such a file system.

So is using another file system.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#213

Post by mattpallissard » Tue, Dec 13, 2022, 7:52 PM UTC

encabulator wrote: post #144

dekhn wrote: post #27

Earlier quoted context omitted.

Many folks I know who manage storage don't make the boot volume RAID (redundant)- instead, it's some rapidly duplicatable thing like an NMVE flash containing the root filesystem, and there's a replacement handy. Then you can bring up and bring the full power of userspace to bear on the RAID repair.

I just finished another late night "thanks" to unknown person who decided to use this approach several years ago while setting up 17 servers that I inherited. The /boot partition of these RHEL machines was put on SD cards, which have been gradually dying. So I got to boot the machine (which is 6000 km away) to recovery mode from ISO and re-create the /boot filesystem on HDD (RAID). Of the 17 servers, only three are n…

This was a trend for a minute back in the late 00's. Any system that was

  1.HA
  2.Had an OS that would get fully loaded into memory (think ESX).

The idea was that SD-cards take less power and where cheap enough you could have a whole gang of spares with images ready to just drop in place. I'm not saying it was a good idea, just that I've encountered this more than once.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#214

Post by mattpallissard » Tue, Dec 13, 2022, 7:55 PM UTC

dekhn wrote: post #27

dale_glass wrote: post #19

But is the RAID handling remotely sane yet? See https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... It has gems such as: * It won't boot on a degraded array by default, requiring manual action to mount it * It won't complain if one of the disks is stale * It won't resilver automatically if a disk is re-added to the array I think the first is the killer. RAID is a High Availability measure. Your system is…

Many folks I know who manage storage don't make the boot volume RAID (redundant)- instead, it's some rapidly duplicatable thing like an NMVE flash containing the root filesystem, and there's a replacement handy. Then you can bring up and bring the full power of userspace to bear on the RAID repair.

I haven't used raid for an OS or boot partition in years. We focus on provisioning and HA. Data goes on a raid an that's it.

Servers go boom, refuse to boot, or even get trigger a few alarms. Disk is replaced, server is booted, and provisioning paves the entire thing faster than you'd be able to debug.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#215

Post by cmurf » Tue, Dec 13, 2022, 7:56 PM UTC

chasil wrote: post #202

cmurf wrote: post #154

Earlier quoted context omitted.

Scrub, not balance. It's primarily a read operation. The on-disk format has all the information needed to do an automatic abbreviated scrub, but the feature isn't implemented yet.

No, I've heard that a scrub will not reallocate missing blocks on mirrors. A rebalance was required in the article below, and this situation really requires attention. ZFS is far friendlier in a crisis. "We'll even manually trigger a scrub—a procedure that storage admins generally understand to look for and automatically repair any data issues... even though we manually initiated a scrub and let it finish, our array…

Scrub checks every data and metadata block and compare to checksum. If missing, or wrong transid, or wrong csum, btrfs will replace the block from a good copy.

There is a potential gotcha with parity being wrong following a crash or powerfail, the so-called write hole. Scrub doesn't check parity, and parity isn't checksummed. The write hole though is arguably two parts: wrong parity and propagated wrong reconstruction.

In the usual write-hole case, both happen silently. Wrong parity could happen after a power fail, crash, misdirected or torn write - whether the array is functioning normally or degraded. However, wrong reconstruction only happens if there's a failure to read a data strip (bad sector or full device failure).

On btrfs, only the wrong parity being written is possible. Upon reconstruction from bad parity, the resulting data is compared to csum and will fail, thus propagation doesn't happen.

To cause parity to be recomputed and rewritten, yes you need to do a full balance. But among all the block group profiles, that's only raid5 and raid6. Single, DUP, raid1, raid1c3, raid1c4, raid10 aren't affected and a scrub does reallocate missing blocks on mirrors.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#216

Post by CoolCold » Tue, Dec 13, 2022, 7:59 PM UTC

whatever1 wrote: post #164

I read all these horror stories here for RAID config and data availability/integrity and I wonder how do the Cloud providers resolve all these issues at their scale? Why bare-metal admins have to spend countless man-hours on troubleshooting these?

You can start with backblaze articles on their data storage tier, will give some hints

Also note, comments on this thread are heavily biased on DYI things and not covering enterprise storage areas, which for instance can come as Blackbox with exported LUNs

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#217

Post by CoolCold » Tue, Dec 13, 2022, 8:07 PM UTC

rtp4me wrote: post #120

sliken wrote: post #87

Earlier quoted context omitted.

Glad it's worked for you. I don't think I've had anything that conflicts with your experiences. I didn't mean to imply a degraded hardware RAID wouldn't mount, I think that's a BTRFS RAID5 issue. Generally I'd consider BTRFS a toy, doubly so for RAID5/6 on BTRFS. Generally I (and colleagues) consider a hardware RAID failure a nightmare and a SAS HBA failure an annoyance. Doubly so if your storage design included cros…

Interesting, thanks for the perspective. Myself, I treat MDADM/ZFS RAID failure more of a headache than RAID cards. As I mentioned earlier, RAID cards allow for easy hot-swap without any OS intervention. I am always nervous when I have to replace drives out of a ZFS pool - mainly because I forget the exact commands that need to be run (in order) for a successful swap. With hardware RAID - no commands :-) Thus far, we…

> mainly because I forget the exact commands that need to be run (in order) for a successful swap. With hardware RAID - no commands :-)

I find it useful to implement some sort of tag or link to wiki be added to alerts information [like on failed drive in your case] - makes much easier to every team member to fix the issue without guessing/invent own way during the outage.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#218

Post by michaelmrose » Tue, Dec 13, 2022, 8:09 PM UTC

timbit42 wrote: post #204

mekster wrote: post #95

Earlier quoted context omitted.

Using Ubuntu is hell of a lot faster than waiting for such a file system.

Ubuntu is embracing an extending Linux through it's Snap repo. Dump it now.

Alternatively just continue ignoring snap until it goes away

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#219

Post by mattpallissard » Tue, Dec 13, 2022, 8:17 PM UTC

I, as many, have a BTRFS raid 5/6 story. I once worked at a place that had double digit TB databases. A few of the servers that predated me were set up with BTRFS raid 10 as their backup strategy. Basically stop the server, snapshot the volume, backup from the snapshot. It worked well enough.

There came a point where we were going to have to move to a larger server as there were no more slots for additional disks. New hardware procured, provisioned, prepped, ready to roll with the exception of an actual cut over. However, c-suite got involved as they tend to do from time to time, and kept kicking the can down the road.

Eventually disk space was a critical issue and a junior sysadmin decided it was a good idea to tell the COO that btrfs can be converted to raid 5 on the fly. I over the phone told them that this would guarantee data loss and refused. They literally came to my office to force my hand. Some people just have to learn the hard way.

That story aside, I'm a big fan of BTRFS and have been using in various configurations on my desktops for years. I'm glad to see this progressing.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#220

Post by CoolCold » Tue, Dec 13, 2022, 8:21 PM UTC

rtp4me wrote: post #120

sliken wrote: post #87

Earlier quoted context omitted.

Glad it's worked for you. I don't think I've had anything that conflicts with your experiences. I didn't mean to imply a degraded hardware RAID wouldn't mount, I think that's a BTRFS RAID5 issue. Generally I'd consider BTRFS a toy, doubly so for RAID5/6 on BTRFS. Generally I (and colleagues) consider a hardware RAID failure a nightmare and a SAS HBA failure an annoyance. Doubly so if your storage design included cros…

Interesting, thanks for the perspective. Myself, I treat MDADM/ZFS RAID failure more of a headache than RAID cards. As I mentioned earlier, RAID cards allow for easy hot-swap without any OS intervention. I am always nervous when I have to replace drives out of a ZFS pool - mainly because I forget the exact commands that need to be run (in order) for a successful swap. With hardware RAID - no commands :-) Thus far, we…

And thanks for the link provided, quite interesting. I plan to redo my canary server with zfs and mysql (primary case for me is ~ 3 times compression) as current one is not much stable highly likely due to my tries to use latest zfs with Ubuntu HWE on 20.04 - this time will use 22.04 with stock ZFS and will go through your findings more closely.