Case study: recovery of a corrupted 12 TB multi-device pool • Hacker News Board

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#71

Post by theamk » Tue, Apr 07, 2026, 2:58 AM UTC

throwaway270925 wrote: post #42

harshreality wrote: post #27

Using DUP as the metadata profile sounds insane. Changing the metadata profile to at least raid1 (raid1, raid1c3, raid1c4) is a good idea, especially for anyone, against recommendations, using raid5 or raid6 for a btrfs array (raid1c3 is more appropriate for raid6). That would make it very difficult for metadata to get corrupted, which is the lion's share of the higher-impact problems with raid5/6 btrfs. check: btrfs…

This should be at the top, using metadata DUP on a 3 disk volume is already asking for it, and of course you loose data when you just use it as jbod with data stored only once. Unless this are enterprise disks with capacitors anything can happen when it suddenly looses power. Not the FSes fault. With the same configuration this can happen with ZFS, bcachefs etc just as well.

> Unless this are enterprise disks with capacitors anything can happen when it suddenly looses power. Not the FSes fault.

Most filesystems just get a few files/directories damaged though. ZFS is famous for handling totally crazy things like broken hardware which damages data in-transit. ext4 has no checksum, but at least fsck will drop things into lost+found directory.

The "making all data inaccessible" part is pretty unique to btrfs, and lets not pretend nothing can be done about this.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#72

Post by singron » Tue, Apr 07, 2026, 6:58 PM UTC

scottlamb wrote: post #60

zootboy wrote: post #55

Earlier quoted context omitted.

> Metadata DUP (not sure if it's across 2 disks or all 3) should be expected to be robust, I'd expect? No. DUP will happily put both copies on the same disk. You would need to use RAID1 (or RAID1c3 for a copy on all disks) if you wanted a guarantee of the metadata being on multiple disks.

Wow, yuck. (The "Why do we even have that lever?!" line comes to mind.) ...even so, without a disk failure, that probably wasn't the cause of this event.

ZFS has similar configurations possible (e.g. copies).

You can end up in this state with btrfs if you start with a single device (defaults to data=single,metadata=dup), and then add additional devices without changing the data/metadata profiles. Or you can choose this config explicitly.

I really wish the btrfs-progs had a --this-config-is-bad-but-continue-anyway flag since there are so many bad configurations possible (raid5/raid6, raid0/single/dup). The rescue tools are also bad and are about as likely to make the problem worse as fix it.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#73

Post by guilhas » Sun, Apr 12, 2026, 4:04 PM UTC

This reads something between cluless/malicious and genius. Crosses several red lights with a car, smashes the car, rebuilds the car with AI, tells people to cross red lights

Keeps repeating btrfs check --repair . This command is dangerous and warned anywhere as a last resort: if you try to execute it you get a warning; the documentation has a warning; any guide from google tell you not to run it unless all else fails; chatgpt/lechat do not metion it, or note it as last resort. So not sure why he keeps repeating it without any note

> Use these tools ONLY if btrfs check --repair segfaults, enters an infinite loop, or leaves the filesystem in worse shape than before.

> Timeline of events ... First repair attempts. btrfs check --repair

The guy is recommending people brick their volumes permanently as first resort without any warning

Between using a dup profile and this I would not be surprised a btrfs dev just disregarding all as slop

> Pool only mounts with rescue=all,ro, fails to mount RW

Also this is important, the data was not lost. Even though read-only

I don't think I would run this code. Still it would be interesting a btrs dev to have look and comment if there is any value in the code generated. As it would be definitely interesting being able to repair more issues in the pool safely inplace