Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

phoronix.com

Post reply on HN
Print view View original

221–230 of 231 posts

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#221

Post by michaelmrose »

imiric wrote: post #48
oarsinsync wrote: post #29

Earlier quoted context omitted.

I did that for a long time. I reached 15 disks. Then something went wrong, and I lost everything. I now run multiple 6 disk raidz2s, in part because I wanted to remove the option that led me down the bad path. I hope your journey ends better than mine did.

Reading these horror stories just makes me not want to rely on smart filesystems at all. If a bug can take out my whole array, or if the system is as inflexible and magical as ZFS, I'd rather not rely on it at all. I've been managing a home NAS for years with SnapRAID and plain old ext4. Sure, it doesn't have the bells and whistles of something like ZFS, but it's simple to use and understand, scales with anything you…

Cars require regular maintenance and have known failure modes if you don't bother to do it its neither magical nor dangerous. Snapraid + ext4 doesn't particularly simpler than zfs just different. The fundamental tradeoff of space vs reliability seems to be mathematically tied. It's impossible to do better in one dimension without sacrificing the other. Making it possible to lose SOME of your data in fact seems to be the worst choice. ZFS makes it trivial to lose nothing. Use enough hardware and do cheap replication regularly and scrub periodically.

If anything the fact that snapraid by not live replicating every change to every disk in the array makes it impossible to achieve the reliability offered by ZFS. It's forever the inferior cousin and in fact more complicated for layering a dissimilar technology on top of the other.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#222

Post by timbit42 »

michaelmrose wrote: post #218
timbit42 wrote: post #204

Earlier quoted context omitted.

Ubuntu is embracing an extending Linux through it's Snap repo. Dump it now.

Alternatively just continue ignoring snap until it goes away

I do, by not using Ubuntu.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#223

Post by AdamJacobMuller »

Scaevolus wrote: post #30
denkmoon wrote: post #21

Earlier quoted context omitted.

Do Meta not develop btrfs for their internal use? I don't think community sentiment is a big factor for them.

Meta and related companies have much better systems available for durability than RAID5. I would be moderately surprised if they commonly use any standard RAID level.

I imagine the "does not boot degraded" was mildly annoying to them, until they figured out the rootfsopts flag and deployed it.

That whole thread above debating not booting degraded really made me lose any faith in btrfs.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#224

Post by AdamJacobMuller »

boardwaalk wrote: post #141

As someone who is in the middle of pulling apart a BTRFS volume by hand (read: writing code to interpret the data structures) to try and recover it, I think being burnt enough is once. No indication of any hardware issue: No recent power loss (& it's on a UPS), no SMART issues, no memory test positives. But the block tree (at least, WIP) is f*cked across all the disks (looks like two competing writers went at it) and…

I've been there, not with BTRFS but with reiserfs many many years ago. I don't envy you.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#225

Post by zeotroph »

mgerdts wrote: post #119
zeotroph wrote: post #104

Earlier quoted context omitted.

Is there a zfs native way to create raid0 disks now? On Linux I usually set up a linear mapping with dmsetup, and then point zfs to that. But this hides the native disks from zfs. I've done that a couple times to make a bunch of mismatched disk fit into zfs's requirement that all are of the same size. The ability to work with a mix of disk sizes is one advantage of btrfs - if you remember to be extra careful on a dis…

> Is there a zfs native way to create raid0 disks now? zpool create swimming disk1 disk2 … That has worked for over 15 years.

Indeed, it is less raid0 but creating a raid0 and then building a raidz on top of it.

Say you have `1, 1, 2, 2` sized devices. A raidz requires them all to be equally sized which can be done by first adding `1 + 1 = 2` and then creating a zpool with `[1+1], 2, 2`. Can this 1 + 1 step be done within zfs? Hypothetically this would be done by creating a raid0 pool and then a raidz on top via `zfs create tank raidz (raid0 a1 b1) c2 d2`, or first running `zfs create ab1 a1 ab` and use that in the next `zfs create ab1 raidz ab1 c2 d2`.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#226

Post by happymellon »

wokkel wrote: post #176
happymellon wrote: post #166

Earlier quoted context omitted.

I've had more issues with my ZFS arrays than the BTRFS ones. The Linux implementation is pretty awful, with it demanding that it uses the absolute `/dev/sdx` reference even if you try to build it using serials or other unchaging reference. Replace a disk and then after the next restart it fails to bring up the RAID because `sdf` points to a different disk. At least BTRFS uses internal UUIDs so can bring up the arrays…

Uh, using /dev/disk/by-id to assemble a zfs array is pretty standard for zol. I never use device names and shuffling disks has never required any manual intervention on my part. Was this a very old ZoL?

This was last year on Ubuntu.

I assembled the zfs array using by-id, but after a reboot zfs reports the drives as being on /dev/sdx.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#227

Post by imiric »

michaelmrose wrote: post #221
imiric wrote: post #48

Earlier quoted context omitted.

Reading these horror stories just makes me not want to rely on smart filesystems at all. If a bug can take out my whole array, or if the system is as inflexible and magical as ZFS, I'd rather not rely on it at all. I've been managing a home NAS for years with SnapRAID and plain old ext4. Sure, it doesn't have the bells and whistles of something like ZFS, but it's simple to use and understand, scales with anything you…

Cars require regular maintenance and have known failure modes if you don't bother to do it its neither magical nor dangerous. Snapraid + ext4 doesn't particularly simpler than zfs just different. The fundamental tradeoff of space vs reliability seems to be mathematically tied. It's impossible to do better in one dimension without sacrificing the other. Making it possible to lose SOME of your data in fact seems to be…

SnapRAID has its drawbacks, sure. As with any technology, deciding to use one solution over another is a balancing act of choosing the set of drawbacks that are acceptable for a particular use case.

In this case, _for my simple needs_ of running a home NAS, the fact SnapRAID doesn't run in real-time is not an issue. In fact, I prefer being in control of when it runs, and what it's doing exactly. Having a short time window where some new data is not replicated is a negligible drawback to me.

OTOH, while ZFS solves this particular issue, its drawbacks of being difficult to scale, and a huge black box that claims to "just work", when in fact my _entire_ array relies on it working perfectly, 100% of the time, are a tough pill to swallow. I'm sure that with the years of dedicated improvements and stability fixes, it's a battle-tested system where the likelihood of it failing is close to 0, even on Linux. But the fact that it's theoretically possible is a deal-breaker to me. In this sense, I much prefer SnapRAID's approach that makes this literally impossible. SnapRAID could stop working entirely or disappear tomorrow, and all my data is perfectly safe.

So, yes, SnapRAID + ext4 is radically simpler than ZFS, IMO.

Would I recommend this setup in a corporate environment, where company resources are on the line? Probably not. But I would still advocate against ZFS and mdadm, and probably suggest something like Ceph instead.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#228

Post by rtp4me »

CoolCold wrote: post #220
rtp4me wrote: post #120

Earlier quoted context omitted.

Interesting, thanks for the perspective. Myself, I treat MDADM/ZFS RAID failure more of a headache than RAID cards. As I mentioned earlier, RAID cards allow for easy hot-swap without any OS intervention. I am always nervous when I have to replace drives out of a ZFS pool - mainly because I forget the exact commands that need to be run (in order) for a successful swap. With hardware RAID - no commands :-) Thus far, we…

And thanks for the link provided, quite interesting. I plan to redo my canary server with zfs and mysql (primary case for me is ~ 3 times compression) as current one is not much stable highly likely due to my tries to use latest zfs with Ubuntu HWE on 20.04 - this time will use 22.04 with stock ZFS and will go through your findings more closely.

Cool! Feel free to reach out if you have any questions (contact details in profile). I am by no means a ZFS expert, but I have been doing servers/storage for a long time and can probably help out if you run into an issue.

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#229

Post by ac29 »

_kdave wrote: post #210
linuxandrew wrote: post #160

Earlier quoted context omitted.

3. Prevent bit-rot w/ software RAID. Running ZFS, btrfs et al. on a single disk can detect bit-rot but not repair it.

By default the metadata profile is 'DUP' (i.e. 2 copies on one device), the same can be done for data but this reduces the usable capacity. On normal HDD or SSD this should not be needed but having both data and metadata DUP/DUP has been useful on raspberry pi with the micro SD card storage. It's not perfect but increases the chances to get the data back if the card is partially damaged due to power spikes.

I have wondered about this myself - does keeping 2 copies of the same data on a single flash-based drive actually increase reliability? Or is the flash controller going to end up combining the two writes into the same block?

Re: Btrfs in Linux 6.2 brings performance improvements, better RAID 5/6 reliability

#230

Post by oarsinsync »

imiric wrote: post #48
oarsinsync wrote: post #29

Earlier quoted context omitted.

I did that for a long time. I reached 15 disks. Then something went wrong, and I lost everything. I now run multiple 6 disk raidz2s, in part because I wanted to remove the option that led me down the bad path. I hope your journey ends better than mine did.

Reading these horror stories just makes me not want to rely on smart filesystems at all. If a bug can take out my whole array, or if the system is as inflexible and magical as ZFS, I'd rather not rely on it at all. I've been managing a home NAS for years with SnapRAID and plain old ext4. Sure, it doesn't have the bells and whistles of something like ZFS, but it's simple to use and understand, scales with anything you…

> will probably move to Ceph ... Every so often I look into btrfs/ZFS/mdadm, but keep reaching the same conclusion that it's just not worth the risk.

The worst thing about my experiences with mdadm and ZFS is that I've had exactly one catastrophic failure in 15 years.

The best thing about my experiences with Ceph is that it's really reinforced the importance of a good backup strategy.

The combination of the two means I now run ZFS, and I have a comprehensive backup strategy that is regularly tested, but has never needed to be invoked.

I strongly recommend developing a strong backup story before you go down your Ceph journey.

Post reply on HN