Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Case study: recovery of a corrupted 12 TB multi-device pool

github.com

Post reply on HN
Print view View original

61–70 of 73 posts

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#61

Post by raron »

fpoling wrote: post #18
phoronixrly wrote: post #2

To theal author: did you continue using btrfs after this ordeal? An FS that will not eat (all) your data upon a hard powercycle only at the cost of 14 custom C tools is a hard pass from me no matter how many distros try to push it down my throat as 'production-ready'... Also, impressive work!

What are the alternatives to btrfs? At 12 TB data checksums are a must unless the data tolerate bit-rot. And if one wants to stick with the official kernel without out-of-tree modules, btrfs is the only choice.

I think you could use dm-integrity over the raw disks to have checksums and protect against bitrot then you can use mdraid to make a RAID1/5/6 of the virtual blockdevs presented by dm-integrity.

I suspect this is still vulnerable to the write hole problem.

You can add LVM to get snapshots, but this still not an end-to-end copy-on-write solution that btrfs and ZFS should provide.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#62

Post by Aachen »

yjftsjthsd-h wrote: post #53
blae wrote: post #50

[flagged]

Why would you assume that people are making up reports of corruption? Is it really inconceivable to you that the thing could have bugs?

> Is it really inconceivable to you that the thing could have bugs?

Or user error, or hardware setups where the docs didn't say "don't do that". If zfs is somehow better in any of those three areas, that would result in fewer corruption stories as well. Hard to know without being able to control for popularity though

Seems really weird to me to assume people make up stories to promote their favorite filesystem. Of course I have one to share as well (opened the thread without knowing it was about btrfs to begin with, I'm not brigading...)

---

I tried btrfs once in my life. I wanted to (1) mirror two disks so a routine disk failure doesn't mean I lose X hours of updates since the last off-site backup, and (2) detect bit rot. And of course it resulted in a giant headache:

The disks got out of sync, put themselves in read-only mode with different data on each (which one has the latest data? Do they both have new fragments?), I eventually figured out which one has the latest data, and I mix up the source and destination device in the recovery command. Iirc the latter was caused by me stopping to read the man page when I found the info I was after and didn't read the whole thing carefully, where subsequent text would have clued me in

The recovery mess-up is user error but if this happens to people on btrfs more often than zfs, maybe zfs is more recommendable anyway. But I've not tried zfs so that's not a statement I can make

I'm back to ext4. Will just use backups and hope for the best. This constant risk of full-filesystem corruption isn't worth it to catch the few files that changed in the last hours, or the few bytes that will rot over my lifetime. On my todo list is writing a little tool that just stores sha2sum+mtime for each file and alerts me if the former changed without the latter, then I can retrieve it from backup and perhaps swap out the disk

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#63

Post by zootboy »

scottlamb wrote: post #60
zootboy wrote: post #55

Earlier quoted context omitted.

> Metadata DUP (not sure if it's across 2 disks or all 3) should be expected to be robust, I'd expect? No. DUP will happily put both copies on the same disk. You would need to use RAID1 (or RAID1c3 for a copy on all disks) if you wanted a guarantee of the metadata being on multiple disks.

Wow, yuck. (The "Why do we even have that lever?!" line comes to mind.) ...even so, without a disk failure, that probably wasn't the cause of this event.

The DUP profile is meant for use with a single disk. The RAID* profiles are meant for use with multiple disks. Both are necessary to cover the full gamut of BTRFS use cases, but it would probably be good if mkfs.btrfs spat out a big warning if you use DUP on a multi-disk filesystem, as this is /usually/ a mistake.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#64

Post by bombela »

stingraycharles wrote: post #37
rcxdude wrote: post #16

Earlier quoted context omitted.

btrfs's reputation is not great in this regard.

As far as I understand, single device and RAID1 is solid, but as soon as you want to do RAID1+0 or RAID5/6 you’re entering dangerous territory with BTRFS.

I had a metadata corruption in metadata raid1c3 (raid1, 3 copies) over 4 disks. It happened after an unplanned power loss during a simulated disk failure replacement. Since manual cleanup of the filesystem metadata (list all files, get IO errors, delete IO errored files), the btrfs kernel driver segfaults in kernel space on any scrub or device replacment attenpt.

Honestly the code of btrfs is a bit scary to read too. I have lost all trust in this filesystem.

Too bad because btrfs has pretty compelling features.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#65

Post by bombela »

harshreality wrote: post #27

Using DUP as the metadata profile sounds insane. Changing the metadata profile to at least raid1 (raid1, raid1c3, raid1c4) is a good idea, especially for anyone, against recommendations, using raid5 or raid6 for a btrfs array (raid1c3 is more appropriate for raid6). That would make it very difficult for metadata to get corrupted, which is the lion's share of the higher-impact problems with raid5/6 btrfs. check: btrfs…

I still got corrupted metadata with metadata raid1c3 on btrfs on a power loss. I never had this happen with ext4 alone or atop Linux raid.

I want to be clear that losing (meta)data in flight during a power loss is expected. But a broken filesystem after that is definitely not acceptable.

Some postgresql db endedup soft corrupted. Postgresql could not replay its log because btrfs threw IO errors on fsync. That's just plain not acceptable.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#66

Post by londons_explore »

Btrfs allows migration from ext4 with a rather good rollback strategy...

Post-migration, a complete disk image of the original ext4 disk will exist within the new filesystem, using no additional disk space due to the magic of copy-on-write.

Why isn't the repair process the same? Fix the filesystem to get everything online asap, and leave a complete disk image of the old damaged filesystem so other recovery processes can be tried if necessary.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#67

Post by scottlamb »

zootboy wrote: post #63
scottlamb wrote: post #60

Earlier quoted context omitted.

Wow, yuck. (The "Why do we even have that lever?!" line comes to mind.) ...even so, without a disk failure, that probably wasn't the cause of this event.

The DUP profile is meant for use with a single disk. The RAID* profiles are meant for use with multiple disks. Both are necessary to cover the full gamut of BTRFS use cases, but it would probably be good if mkfs.btrfs spat out a big warning if you use DUP on a multi-disk filesystem, as this is /usually/ a mistake.

[dead]

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#68

Post by phoronixrly »

fpoling wrote: post #57
phoronixrly wrote: post #35

Earlier quoted context omitted.

I said lvmraid [1][2]. [1] https://www.man7.org/linux/man-pages/man7/lvmraid.7.html#DAT... [2] https://docs.redhat.com/en/documentation/red_hat_enterprise_...

But that is just raid on top of dm-integrity. And Redhat docs omits an important part when suggesting to use the bitmap mode with dm-integrity: man 8 integritysetup: --integrity-bitmap-mode. -B Use alternate bitmap mode (available since Linux kernel 5.2) where dm-integrity uses bitmap instead of a journal. If a bit in the bitmap is 1, then corresponding region’s data and integrity tags are not synchronized - if the m…

> But that is just raid on top of dm-integrity

As I said -- boring tech. Just what I like when not in a mood to lose data.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#69

Post by abrookewood »

curt15 wrote: post #31
stinkbeetle wrote: post #3

> Case study: recovery of a severely corrupted 12 TB multi-device pool, plus constructive gap analysis and reference tool set #1107 Please don't be btrfs please don't be btrfs please don't be btrfs...

Where are all of the ZFS corruption stories? Or are there simply fewer of those?

I love ZFS, but have been corrected a couple of times when I said it was bomb proof. Can't remember the details, but it has served me faithfully for 10 years or so? Plus the bugs were pretty niche if I recall correctly.

Edit: found some comments below: ZFS on Linux has had many bugs over the years, notably with ZFS-native encryption and especially sending/receiving encrypted volumes. Another issue is that using swap on ZFS is still guaranteed to hang the kernel in low memory scenarios, because ZFS needs to allocate memory to write to swap.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#70

Post by abrookewood »

blae wrote: post #50

[flagged]

I am a complete ZFS fanboy, but that doesn't mean I want BTRFS to fail. Competition is healthy and I don't think Oracle is going to change the licence on ZFS any time soon, so it's good to have an option that can be used without encumbrance.
Post reply on HN