Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Case study: recovery of a corrupted 12 TB multi-device pool

github.com

Post reply on HN
Print view View original

21–30 of 73 posts

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#21

Post by Joel_Mckay »

fpoling wrote: post #18
phoronixrly wrote: post #2

To theal author: did you continue using btrfs after this ordeal? An FS that will not eat (all) your data upon a hard powercycle only at the cost of 14 custom C tools is a hard pass from me no matter how many distros try to push it down my throat as 'production-ready'... Also, impressive work!

What are the alternatives to btrfs? At 12 TB data checksums are a must unless the data tolerate bit-rot. And if one wants to stick with the official kernel without out-of-tree modules, btrfs is the only choice.

Could try ZFS or CephFS... even if several host roles are in VM containers (45Drives has a product setup that way.)

The btrfs solution has a mixed history, and had a lot of the same issues DRBD could get. They are great until some hardware/kernel-mod eventually goes sideways, and then the auto-heal cluster filesystems start to make a lot more sense. Note, with cluster based complete-file copy/repair object features the damage is localized to single files at worst, and folks don't have to wait 3 days to bring up the cluster on a crash.

Best of luck, =3

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#22

Post by aktau »

fpoling wrote: post #18
phoronixrly wrote: post #2

To theal author: did you continue using btrfs after this ordeal? An FS that will not eat (all) your data upon a hard powercycle only at the cost of 14 custom C tools is a hard pass from me no matter how many distros try to push it down my throat as 'production-ready'... Also, impressive work!

What are the alternatives to btrfs? At 12 TB data checksums are a must unless the data tolerate bit-rot. And if one wants to stick with the official kernel without out-of-tree modules, btrfs is the only choice.

I tried btrfs on three different occasions. Three times it managed to corrupt itself. I'll admit I was too enthousiastic the first time, trying it less than a year after it appeared in major distros. But the latter two are unforgiveable (I had to reinstall my mom's laptop).

I've been using ZFS for my NAS-like thing since then. It's been rock solid ().

(): I know about the block cloning bug, and the encryption bug. Luckily I avoided those (I don't tend to enable new features like block cloning, and I didn't have an encrypted dataset at the time). Still, all in all it's been really good in comparison to btrfs.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#23

Post by ThatPlayer »

egorfine wrote: post #20
fpoling wrote: post #18

Earlier quoted context omitted.

What are the alternatives to btrfs? At 12 TB data checksums are a must unless the data tolerate bit-rot. And if one wants to stick with the official kernel without out-of-tree modules, btrfs is the only choice.

> if one wants to stick with the official kernel without out-of-tree modules I wonder how could a requirement like that possibly arise. Especially with an obvious exception for zfs.

Bcachefs also fulfills the requirement of checksums (and multi device support).

Also out of tree.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#24

Post by phoronixrly »

fpoling wrote: post #18
phoronixrly wrote: post #2

To theal author: did you continue using btrfs after this ordeal? An FS that will not eat (all) your data upon a hard powercycle only at the cost of 14 custom C tools is a hard pass from me no matter how many distros try to push it down my throat as 'production-ready'... Also, impressive work!

What are the alternatives to btrfs? At 12 TB data checksums are a must unless the data tolerate bit-rot. And if one wants to stick with the official kernel without out-of-tree modules, btrfs is the only choice.

lvm offers lvmraid, integrity, and snapshots as one example. It's old unsexy tech, but losing data is not to my taste lately...

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#25

Post by phoronixrly »

ThatPlayer wrote: post #23
egorfine wrote: post #20

Earlier quoted context omitted.

> if one wants to stick with the official kernel without out-of-tree modules I wonder how could a requirement like that possibly arise. Especially with an obvious exception for zfs.

Bcachefs also fulfills the requirement of checksums (and multi device support). Also out of tree.

Does it not also eat data though?

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#26

Post by [deleted] »

fpoling wrote: post #18
phoronixrly wrote: post #2

To theal author: did you continue using btrfs after this ordeal? An FS that will not eat (all) your data upon a hard powercycle only at the cost of 14 custom C tools is a hard pass from me no matter how many distros try to push it down my throat as 'production-ready'... Also, impressive work!

What are the alternatives to btrfs? At 12 TB data checksums are a must unless the data tolerate bit-rot. And if one wants to stick with the official kernel without out-of-tree modules, btrfs is the only choice.

[deleted]

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#27

Post by harshreality »

Using DUP as the metadata profile sounds insane.

Changing the metadata profile to at least raid1 (raid1, raid1c3, raid1c4) is a good idea, especially for anyone, against recommendations, using raid5 or raid6 for a btrfs array (raid1c3 is more appropriate for raid6). That would make it very difficult for metadata to get corrupted, which is the lion's share of the higher-impact problems with raid5/6 btrfs.

check:

    btrfs fi df
convert metadata:

    btrfs balance start -mconvert=raid1c3,soft
(make sure it's -mconvert — m is for metadata — not -dconvert which would switch profiles for data, messing up your array)

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#29

Post by stinkbeetle »

fpoling wrote: post #18
phoronixrly wrote: post #2

To theal author: did you continue using btrfs after this ordeal? An FS that will not eat (all) your data upon a hard powercycle only at the cost of 14 custom C tools is a hard pass from me no matter how many distros try to push it down my throat as 'production-ready'... Also, impressive work!

What are the alternatives to btrfs? At 12 TB data checksums are a must unless the data tolerate bit-rot. And if one wants to stick with the official kernel without out-of-tree modules, btrfs is the only choice.

What devices are you talking about, what's the UBER, over what period of time?

RAID and logical block redundancy has scaled to petabytes for years in serious production use, before btrfs was even developed.

Post reply on HN