Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

phoronix.com

Post reply on HN
Print view View original

161–170 of 231 posts

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#161

Post by justsomehnguy »

wtallis wrote: post #159
justsomehnguy wrote: post #151

Earlier quoted context omitted.

> Who reboots a production system without monitoring it? Anyone. > Also if you boot with a degraded disk are you not asking for massive trouble > not being able to boot until you add another disk Great. I'm just a [sys]admin who were given a task to do something on $server. I jump around the red tape, claw out a 15 minute downtime because the task requires reboot, proceed with all that corporate dance with notificati…

It sounds like you're hypothesizing a long chain of bad decisions, and then ridiculing btrfs for taking the choice that means your next bad decision only culminates in (predictable, preventable) downtime rather than data loss.

> It sounds like you're hypothesizing

This is all the things what I encountered in my admin days.

Including BL670 with incorrectly connected drives, so despite everything saying (and indicating) what the failed drive was in bay 2 it was actually in bay 1.

> then ridiculing btrfs

I ridicule btrfs for it's RAID mode not being a RAID mode by default.

RAID is about availability of data.

If you so hell bent on the data safety then btfrs should kernel panic as soon as one drive degrades. THAT would make sure someone would come and investigate what happened and no data loss.

Right?

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#162

Post by happymellon »

justsomehnguy wrote: post #158
wtallis wrote: post #156

Earlier quoted context omitted.

Booting from a degraded array is only a fine idea in some circumstances, not all. That's why the kernel should not default to automatically doing so; but a distro or sysadmin that has better knowledge of the broader situation (eg. presence of hot spares or a working monitoring/alert system) can reasonably change that default when the risks of booting from a degraded array have been mitigated.

Ie you are treating RAID as a backup.

No, you are treating RAID as a protection against longer outage of restoring from backups.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#163

Post by wtallis »

justsomehnguy wrote: post #158
wtallis wrote: post #156

Earlier quoted context omitted.

Booting from a degraded array is only a fine idea in some circumstances, not all. That's why the kernel should not default to automatically doing so; but a distro or sysadmin that has better knowledge of the broader situation (eg. presence of hot spares or a working monitoring/alert system) can reasonably change that default when the risks of booting from a degraded array have been mitigated.

Ie you are treating RAID as a backup.

Backups cannot be perfectly real-time unless they are very nearly RAID. Any time you are generating/collecting important data, you will unavoidably have some amount of that important data in the state of not yet backed up.

It's reasonable to want to preserve all the data you currently have—some of which probably hasn't been backed up yet—and not accept new data to be written with the durability guarantees the array was originally configured for silently violated.

Since the kernel has no way of knowing which volumes may contain important data that didn't get the chance to be backed up, it should try its best to maintain the original durability standards the filesystem was configured until some mechanism outside the kernel authorizes the relaxation of those standards.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#164

Post by whatever1 »

I read all these horror stories here for RAID config and data availability/integrity and I wonder how do the Cloud providers resolve all these issues at their scale?

Why bare-metal admins have to spend countless man-hours on troubleshooting these?

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#165

Post by happymellon »

klysm wrote: post #24
dale_glass wrote: post #19

But is the RAID handling remotely sane yet? See https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... It has gems such as: * It won't boot on a degraded array by default, requiring manual action to mount it * It won't complain if one of the disks is stale * It won't resilver automatically if a disk is re-added to the array I think the first is the killer. RAID is a High Availability measure. Your system is…

> * It won't boot on a degraded array by default, requiring manual action to mount it That by itself is a complete deal breaker.

Unless you give it permission.

It's an option to enable. If you needing to configure something is a dealbreaker then having RAID is probably not for you anyway.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#166

Post by happymellon »

jiggawatts wrote: post #75
dale_glass wrote: post #19

But is the RAID handling remotely sane yet? See https://arstechnica.com/gadgets/2021/09/examining-btrfs-linu... It has gems such as: * It won't boot on a degraded array by default, requiring manual action to mount it * It won't complain if one of the disks is stale * It won't resilver automatically if a disk is re-added to the array I think the first is the killer. RAID is a High Availability measure. Your system is…

The list of bugs and counter-intuitive design decisions in BTRFS RAID should make anyone pause before ever considering it as a viable filesystem for anything . It's like people justifying MySQL saying that it's okay for it to lose or corrupt data, and that transactional integrity doesn't matter as much as people say it does. Yeah, maybe if you're website is a blog. But anyone storing real data should run away screami…

I've had more issues with my ZFS arrays than the BTRFS ones.

The Linux implementation is pretty awful, with it demanding that it uses the absolute `/dev/sdx` reference even if you try to build it using serials or other unchaging reference.

Replace a disk and then after the next restart it fails to bring up the RAID because `sdf` points to a different disk. At least BTRFS uses internal UUIDs so can bring up the arrays when the disk ids change.

Fairly fundamental stuff, rather than complaining about an option not being enabled by default.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#167

Post by StreamBright »

encabulator wrote: post #144
dekhn wrote: post #27

Earlier quoted context omitted.

Many folks I know who manage storage don't make the boot volume RAID (redundant)- instead, it's some rapidly duplicatable thing like an NMVE flash containing the root filesystem, and there's a replacement handy. Then you can bring up and bring the full power of userspace to bear on the RAID repair.

I just finished another late night "thanks" to unknown person who decided to use this approach several years ago while setting up 17 servers that I inherited. The /boot partition of these RHEL machines was put on SD cards, which have been gradually dying. So I got to boot the machine (which is 6000 km away) to recovery mode from ISO and re-create the /boot filesystem on HDD (RAID). Of the 17 servers, only three are n…

So you discovered the well known problem in "IoT" that SD cards should never be used for filesystems that are online 100% of the time?

I think the parent views on storage is still valid. Do not boot from the RAID volume where you store many terabytes.

I usually have a small mirror SSD for the OS (like 128G or max 256G). Those drives are super cheap, easy to replace.

For the /data I usually use ZFS or mdadm + xfs. Depending on the use case (for example do you want to add more capacity later? ) you can decide.

Using SDcards for server filesystem is negligent at the very least.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#168

Post by wtallis »

justsomehnguy wrote: post #161
wtallis wrote: post #159

Earlier quoted context omitted.

It sounds like you're hypothesizing a long chain of bad decisions, and then ridiculing btrfs for taking the choice that means your next bad decision only culminates in (predictable, preventable) downtime rather than data loss.

> It sounds like you're hypothesizing This is all the things what I encountered in my admin days. Including BL670 with incorrectly connected drives, so despite everything saying (and indicating) what the failed drive was in bay 2 it was actually in bay 1. > then ridiculing btrfs I ridicule btrfs for it's RAID mode not being a RAID mode by default . RAID is about availability of data. If you so hell bent on the data s…

> If you so hell bent on the data safety then btfrs should kernel panic as soon as one drive degrades. THAT would make sure someone would come and investigate what happened and no data loss.

> Right?

Panicking an already-running kernel would only serve to prevent userspace from handling the failure through mechanisms that are inherently beyond the scope and capabilities of the kernel alone (ie. stuff like alerting a sysadmin, activating a hot spare, or initiating a rebalance to restore redundancy among the remaining drives).

Perhaps the kernel should default to freezing a non-root filesystem when it becomes degraded, absent an explicit configuration permitting otherwise. But for the root filesystem, that would be counterproductive and prevent the failure from being handled gracefully.

Obviously, the tradeoffs are different for a system that is still trying to boot as opposed to one that is fully up and running.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#169

Post by justsomehnguy »

wtallis wrote: post #163
justsomehnguy wrote: post #158

Earlier quoted context omitted.

Ie you are treating RAID as a backup.

Backups cannot be perfectly real-time unless they are very nearly RAID. Any time you are generating/collecting important data, you will unavoidably have some amount of that important data in the state of not yet backed up . It's reasonable to want to preserve all the data you currently have—some of which probably hasn't been backed up yet—and not accept new data to be written with the durability guarantees the array…

> It's reasonable to want to preserve all the data you currently have—some of which probably hasn't been backed up yet—and not accept new data to be written with the durability guarantees the array was originally configured for silently violated.

IE (by your logic) the system should stop the writes as soon as the array became degraded.

But this is not what happens with btrfs: it would happily continue to write the data on the array until reboot.

And then suddenly it's "oh my god array is degraded!!!111 you should not write to it1111".

To add on that: I never seen for a HW RAID card to stop booting by a mere degraded state of the array. Changes in configuration of arrays, loss of more than enough for redundancy drives - yes, that would halt the boot and require the operator intervention. Array in a degraded state? Just spit the warnings to the console and boot. Nobody has the time to walk to each server with a degraded array on every reboot.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#170

Post by BatteryMountain »

whatever1 wrote: post #164

I read all these horror stories here for RAID config and data availability/integrity and I wonder how do the Cloud providers resolve all these issues at their scale? Why bare-metal admins have to spend countless man-hours on troubleshooting these?

BSD + ZFS
Post reply on HN