Live data from Hacker News

Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

phoronix.com

121–130 of 231 posts

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#121

Earlier quoted context omitted.

Only time I've lost data with zfs was when i was first doing something supremely stupid, mostly as a dare to see if it would work. I had recently upgraded my pool and had a bunch of old disks lying around so i fired them up in a new pool, 6x 4TB and 6x 8TB disks. I took the 4TB disks and set them in in a raid0 to get effectively 9x 8TB disks. Apparently doing this will work but you MUST make sure you properly wipe al…

Is there a zfs native way to create raid0 disks now? On Linux I usually set up a linear mapping with dmsetup, and then point zfs to that. But this hides the native disks from zfs. I've done that a couple times to make a bunch of mismatched disk fit into zfs's requirement that all are of the same size. The ability to work with a mix of disk sizes is one advantage of btrfs - if you remember to be extra careful on a dis…

I believe you can accomplish this with partitioning the disks with sizes that match the smallest disk too for zfs. You still have all the same reliability issues though and there's probably some kimd of performance hit too. Though yoy could then likely set copies=2 or something to add some redundancy to bad sectors at least and get a little better there than the linear mapping with dm or lvm2 on linux. You should the. Also be able to grow the array but like the linear mapping it's going to be unbalanced as to where free space is.

My personal preference in this kind of situation is to still just use lvm2 to jbod the disks together if I;m fine with the risks of losing it all anyway, and just setup a good backup strategy. More because I'm familiar with it and all the tools than other setups. I think zfs might give me more early warning signs of a disk with an issue but the performance hit isn't always worth it to me there. I've got an 8tb raid0 nvme ssd array (4x2tb) setup to give me some blazing fast local storage for a few key VMs thst I just backup daily to the zfs storage. I can get 7.5GB/s reads and 6.7GB/s writes to that array doing that and using an lvm-thin pool so I can nicely snapshot during backups.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#122
post #86
post #27

Earlier quoted context omitted.

Many folks I know who manage storage don't make the boot volume RAID (redundant)- instead, it's some rapidly duplicatable thing like an NMVE flash containing the root filesystem, and there's a replacement handy. Then you can bring up and bring the full power of userspace to bear on the RAID repair.

The way I used to do it (back then getting grub/initrd raid aware wasn't really doable - not sure about now) with mdadm was to use RAID1 on the boot volume such that a disk from a broken mirror was still usable by itself. Other volumes were RAID 5 or 10 etc.

How would a mirrored boot work in practice? Normally the first bootable partition gets loaded if it’s corrupt, it would keep failing at the same bad sectors.

Would it randomly pick one of the bootable disks? That way you have an n-1/n chance of avoiding the bad disk?

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#123
post #24

Earlier quoted context omitted.

> * It won't boot on a degraded array by default, requiring manual action to mount it That by itself is a complete deal breaker.

IMO, the inconvenience this causes is outweighed by the prevention of data loss for people who otherwise would never have noticed the degraded array.

Right but if it's remote server you're fucked and you can't login on it to even assess what happened

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#124

Earlier quoted context omitted.

> The only thing it's missing before I consider it full-featured is stable RAID 5/6. But it looks like that hasn't been forgotten. I've been running BTRFS RAID6 since 2016 and have only had one issue (arch kernel needed to be rolled back) and never suffered anything catastrophic. It's perfectly happy humming along with a 15x8TB raid array.

Are you using Raid6 for both data and metadata? If yes, you could just as well have been running raid 0 and having more space and performance out of the same disks. Because of well documented issues, metadata on raid6 is not yet safe in power/hw failure situations. And if you have no safety in those situations, what is raid even for? (It's perfectly safe to run metadata on raid1 and data on raid5/6.)

I'm running RAID6 for data and RAID1 for metadata. Wanted the metadata replicated. So far I've had no HW failure situations and I have suffered a few power outages while I was away from home even though I have a backup battery.

Although these drives have surpassed 50,000 hours of running time, I am fairly certain the day of reckoning is coming, but I have backups and a plan to recover.

I may recover to something not BTRFS. Not sure just yet.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#125
post #123

Earlier quoted context omitted.

IMO, the inconvenience this causes is outweighed by the prevention of data loss for people who otherwise would never have noticed the degraded array.

Right but if it's remote server you're fucked and you can't login on it to even assess what happened

Don't servers tend to include some kind of out-of-band management like HP iLO, Dell iDRAC, or OpenBMC?

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#126
post #122
post #86

Earlier quoted context omitted.

The way I used to do it (back then getting grub/initrd raid aware wasn't really doable - not sure about now) with mdadm was to use RAID1 on the boot volume such that a disk from a broken mirror was still usable by itself. Other volumes were RAID 5 or 10 etc.

How would a mirrored boot work in practice? Normally the first bootable partition gets loaded if it’s corrupt, it would keep failing at the same bad sectors. Would it randomly pick one of the bootable disks? That way you have an n-1/n chance of avoiding the bad disk?

Back then, grub and initrd didn't understand mdadm so would be set to boot off one of the boot vols as a raw disk. Once initrd handed over to the root vol, it would understand mdadm properly.

If a disk with a boot vol failed, it would have a 50/50 chance of still booting depending on which one failed. You'd create a backup grub entry to boot off the other one manually.

Not as transparent as hardware RAID, and maybe now grub is aware enough to auto handle mdadm? I don't know - I haven't done bare metal mdadm for a decade or so.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#127

Maybe I'm just cynical but I think the ship has sailed for BTRFS RAID 5/6. It's now part of the global mindshare that BTRFS RAID 5/6 == data loss, no one wants to be the guinea pig that proves it works. Better to direct resources towards bcachefs or ZFS IMO.

Ideally we'd be working based on actual information rather than global mindshare. There will be guinea pigs to test it. There are already quite a few people running it despite the warnings. As much as I want to see a wide use of bcachefs, it's still years away. As someone who actually wants to store data - why would you direct resources to bcachefs which is known experimental, rather than btrfs which plainly document…

Right but "actual information" here is "developers over and over fail to nail what ZFS and MDRAID did well decades ago". There is no reason to trust their next try will be "the one"

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#128

Earlier quoted context omitted.

extraneous files seroed Though I've used XFS a lot over the years. Mostly because the Debian installer gave 12yo me the choice between ext2, ext3 and xfs, so XFS it was because it sounded cooler.

I've had pacman and ext4 zero a whole bunch of system files on a forced shutdown during updates. Reminded me of XFS back when.

Come to think of it vast majority of fs voes was ext3/4 family, I've seen zeroed file, I've also seen file replaced by content of another file on unclean shutdowns

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#129
post #70
post #34

Earlier quoted context omitted.

Generally every recommendation I've heard is don't use hardware RAID. Some of the problems: * RAID cards usually have under powered CPUs, and can easily be a bottleneck. Often the best performance with a RAID card is with the RAID disabled. * Even battery backed RAM is often pretty slow and on the wrong end of a high latency connection between CPU/RAM and disks. Generally investments in ram or intentlog/writelog/slog…

My experience with hardware RAID cards (LSI, PERC, etc) is vastly different than the recommendations you heard. I have been running production servers with LSI hardware RAID cards for the past 15 years and have not experienced the items you note. The only thing I did experience was a bad RAID card (solved by replacing it with a similar LSI model). To contrast your "problem" list: * Compared with MDADM (mirrors, RAID-…

I run my two LSI cards in HBA mode. The initial minor annoyance that I had was having to flash the cards in. Which coincidentally was also my first time ever doing something like that.

Do you have any experience with hardware RAID provided by motherboard manufacturers like Asus? I have a spare dual xeon system I may utilize for my next NAS build and I'll pair it with two more RAID cards that can be put in HBA.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#130
I have used btrfs for a low memory footprint raid 0 solution to span drives. Mind you for expendable datasets. It has worked really well, never had failures due to btrfs shall we say -rot.

Filesystem is light on requirements and works really well. Compression works great too.

The only thing I haven't played around with are reflinks.

For critical datasets I apply 3-2-1 backup strategy and use ZFS.

Post reply on HN