Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Case study: recovery of a corrupted 12 TB multi-device pool

github.com

Post reply on HN
Print view View original

51–60 of 73 posts

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#51

Post by scottlamb »

throwaway270925 wrote: post #43

> A hard power cycle on a 3 device pool (data single, metadata DUP, DM-SMR disks) left the extent tree and free space tree in a state that no native repair path could resolve. As a ZFS wrangler by day: People in this thread seem to happily shit on btrfs here but this seems to be very much not like a sane, resilient configuration no matter the FS. Just something to keep in mind.

Might be true, but I don't see any aspect of that which is relevant to this event:

* Data single obviously means losing a single drive will cause data loss, but no drive was actually lost, right?

* Metadata DUP (not sure if it's across 2 disks or all 3) should be expected to be robust, I'd expect?

* I certainly eye DM-SMR disks with suspicion in general, but it doesn't sound like they were responsible for the damage: "Both DUP copies of several metadata blocks were written with inconsistent parent and child generations."

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#52

Post by salt4034 »

Retr0id wrote: post #7

This is obviously LLM output, but perhaps LLM output that corresponds to a real scenario. It's plausible that Claude was able to autonomously recover a corrupted fs, but I would not trust its "insights" by default. I'd love to see a btrfs dev's take on this!

See this Reddit post for background: https://www.reddit.com/r/ClaudeAI/comments/1sdabux/hats_off_...

TLDR: The user got his filesystem corrupted on a forced reboot; native btrfs tools made the failure worse; the user asked Claude to autonomously debug and fix the problem; after multiple days of debugging, Claude wrote a set of custom low-level C scripts to recover 99.9% of the data; the user was impressed and asked Claude to submit an issue describing the whole thing.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#54

Post by Retr0id »

salt4034 wrote: post #52
Retr0id wrote: post #7

This is obviously LLM output, but perhaps LLM output that corresponds to a real scenario. It's plausible that Claude was able to autonomously recover a corrupted fs, but I would not trust its "insights" by default. I'd love to see a btrfs dev's take on this!

See this Reddit post for background: https://www.reddit.com/r/ClaudeAI/comments/1sdabux/hats_off_... TLDR: The user got his filesystem corrupted on a forced reboot; native btrfs tools made the failure worse; the user asked Claude to autonomously debug and fix the problem; after multiple days of debugging, Claude wrote a set of custom low-level C scripts to recover 99.9% of the data; the user was impressed and asked C…

Good to know that my claude-dar is still working.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#55

Post by zootboy »

scottlamb wrote: post #51
throwaway270925 wrote: post #43

> A hard power cycle on a 3 device pool (data single, metadata DUP, DM-SMR disks) left the extent tree and free space tree in a state that no native repair path could resolve. As a ZFS wrangler by day: People in this thread seem to happily shit on btrfs here but this seems to be very much not like a sane, resilient configuration no matter the FS. Just something to keep in mind.

Might be true, but I don't see any aspect of that which is relevant to this event: * Data single obviously means losing a single drive will cause data loss, but no drive was actually lost, right? * Metadata DUP (not sure if it's across 2 disks or all 3) should be expected to be robust, I'd expect? * I certainly eye DM-SMR disks with suspicion in general, but it doesn't sound like they were responsible for the damage:…

> Metadata DUP (not sure if it's across 2 disks or all 3) should be expected to be robust, I'd expect?

No. DUP will happily put both copies on the same disk. You would need to use RAID1 (or RAID1c3 for a copy on all disks) if you wanted a guarantee of the metadata being on multiple disks.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#57

Post by fpoling »

phoronixrly wrote: post #35
fpoling wrote: post #32

Earlier quoted context omitted.

lvm only supports checksums for metadata. It does not checksum the data itself. For checksums with arbitrary filesystems one can have dm-integrity device rather than LVM. But the performance suffer due to separated journal writes by the device.

I said lvmraid [1][2]. [1] https://www.man7.org/linux/man-pages/man7/lvmraid.7.html#DAT... [2] https://docs.redhat.com/en/documentation/red_hat_enterprise_...

But that is just raid on top of dm-integrity. And Redhat docs omits an important part when suggesting to use the bitmap mode with dm-integrity:

man 8 integritysetup:

       --integrity-bitmap-mode. -B
           Use alternate bitmap mode (available since Linux kernel 5.2) where dm-integrity uses bitmap instead of a journal. If a bit in the bitmap is 1, then corresponding region’s data and integrity tags are not synchronized - if the machine crashes, the unsynchronized regions will be recalculated. The bitmap mode is faster than the journal mode, because we don’t have to write the data twice, but it is also less reliable, because if data corruption happens when the machine crashes, it may not be detected.
I just do not see how without a direct filesystem support one can have both reliable checksums and performance.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#59

Post by simoncion »

aktau wrote: post #22
fpoling wrote: post #18

Earlier quoted context omitted.

What are the alternatives to btrfs? At 12 TB data checksums are a must unless the data tolerate bit-rot. And if one wants to stick with the official kernel without out-of-tree modules, btrfs is the only choice.

I tried btrfs on three different occasions. Three times it managed to corrupt itself. I'll admit I was too enthousiastic the first time, trying it less than a year after it appeared in major distros. But the latter two are unforgiveable (I had to reinstall my mom's laptop). I've been using ZFS for my NAS-like thing since then. It's been rock solid ( ). ( ): I know about the block cloning bug, and the encryption bug.…

Additional anecdata:

I've been using btrfs as the primary FS for my laptop for nearly twenty years, and for my desktop and multipurpose box for as long as they've existed (~eight and ~three years, respectively). I haven't had troubles with the laptop FS in like fifteen years, and have never had troubles with the desktop or multipurpose box.

I also used btrfs as the production FS for the volume management in our CI at $DAYJOB, as it was way faster than overlayfs. No problems there, either.

Go figure, I guess.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#60

Post by scottlamb »

zootboy wrote: post #55
scottlamb wrote: post #51

Earlier quoted context omitted.

Might be true, but I don't see any aspect of that which is relevant to this event: * Data single obviously means losing a single drive will cause data loss, but no drive was actually lost, right? * Metadata DUP (not sure if it's across 2 disks or all 3) should be expected to be robust, I'd expect? * I certainly eye DM-SMR disks with suspicion in general, but it doesn't sound like they were responsible for the damage:…

> Metadata DUP (not sure if it's across 2 disks or all 3) should be expected to be robust, I'd expect? No. DUP will happily put both copies on the same disk. You would need to use RAID1 (or RAID1c3 for a copy on all disks) if you wanted a guarantee of the metadata being on multiple disks.

Wow, yuck. (The "Why do we even have that lever?!" line comes to mind.)

...even so, without a disk failure, that probably wasn't the cause of this event.

Post reply on HN