Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Case study: recovery of a corrupted 12 TB multi-device pool

github.com

Post reply on HN
Print view View original

1–10 of 73 posts

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#2

Post by phoronixrly »

To theal author: did you continue using btrfs after this ordeal? An FS that will not eat (all) your data upon a hard powercycle only at the cost of 14 custom C tools is a hard pass from me no matter how many distros try to push it down my throat as 'production-ready'...

Also, impressive work!

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#4

Post by yjftsjthsd-h »

> This is not a bug report. [...] The goal is constructive, not a complaint.

Er, I appreciate trying to be constructive, but in what possible situation is it not a bug that a power cycle can lose the pool? And if it's not technically a "bug" because BTRFS officially specifies that it can fail like that, why is that not in big bold text at the start of any docs on it? 'Cuz that's kind of a big deal for users to know.

EDIT: From the longer write-up:

> Initial damage. A hard power cycle interrupted a commit at generation 18958 to 18959. Both DUP copies of several metadata blocks were written with inconsistent parent and child generations.

Did the author disable safety mechanisms for that to happen? I'm coming from being more familiar with ZFS, but I would have expected BTRFS to also use a CoW model where it wasn't possible to have multiple inconsistent metadata blocks in a way that didn't just revert you to the last fully-good commit. If it does that by default but there's a way to disable that protection in the name of improving performance, that would significantly change my view of this whole thing.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#5

Post by rincebrain »

yjftsjthsd-h wrote: post #4

> This is not a bug report. [...] The goal is constructive, not a complaint. Er, I appreciate trying to be constructive, but in what possible situation is it not a bug that a power cycle can lose the pool? And if it's not technically a "bug" because BTRFS officially specifies that it can fail like that, why is that not in big bold text at the start of any docs on it? 'Cuz that's kind of a big deal for users to know.…

As far as I can see, no, the author disabled nothing of the sort that he documented.

I suspect that the author's intent is less "I do not view this as a bug" and more "I do not think it's useful to get into angry debates over whether something is a bug". I do not know whether this is a common thing on btrfs discussions, but I have certainly seen debates to that effect elsewhere.

(My personal favorite remains "it's not a data loss bug if someone could technically theoretically write something to recover the data". Perhaps, technically, that's true, but if nobody is writing such a tool, nobody is going to care about the semantics there.)

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#7

Post by Retr0id »

This is obviously LLM output, but perhaps LLM output that corresponds to a real scenario. It's plausible that Claude was able to autonomously recover a corrupted fs, but I would not trust its "insights" by default. I'd love to see a btrfs dev's take on this!

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#8

Post by number6 »

Retr0id wrote: post #7

This is obviously LLM output, but perhaps LLM output that corresponds to a real scenario. It's plausible that Claude was able to autonomously recover a corrupted fs, but I would not trust its "insights" by default. I'd love to see a btrfs dev's take on this!

This is also my first impulse. The second was, if this happened to me, I would not be able to recover it. All the custom c tool talk... If you ask Claude Code it will code something up.

Well that he recovered the disks is amazing in itself. I would have given up and just pulled a backup.

However, I would like to see a Dev saying: why didn't you use the -- which we created for this Usecase

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#9

Post by yjftsjthsd-h »

Retr0id wrote: post #7

This is obviously LLM output, but perhaps LLM output that corresponds to a real scenario. It's plausible that Claude was able to autonomously recover a corrupted fs, but I would not trust its "insights" by default. I'd love to see a btrfs dev's take on this!

I was assuming real scenario with heavy LLM help to recover. Would be nice for the author to clarify. And, separately, for BTRFS devs to weigh in, though I'd somewhat prefer to get some indication that it's real before spending their time.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#10

Post by Retr0id »

yjftsjthsd-h wrote: post #4

> This is not a bug report. [...] The goal is constructive, not a complaint. Er, I appreciate trying to be constructive, but in what possible situation is it not a bug that a power cycle can lose the pool? And if it's not technically a "bug" because BTRFS officially specifies that it can fail like that, why is that not in big bold text at the start of any docs on it? 'Cuz that's kind of a big deal for users to know.…

Unless I missed it the writeup never identifies a causal bug, only things that made recovery harder.
Post reply on HN