Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Case study: recovery of a corrupted 12 TB multi-device pool

github.com

Post reply on HN
Print view View original

41–50 of 73 posts

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#41

Post by nubinetwork »

praseodym wrote: post #14
yjftsjthsd-h wrote: post #13

Earlier quoted context omitted.

It could be ZFS. I'd be much more surprised, but it can still have bugs.

ZFS on Linux has had many bugs over the years, notably with ZFS-native encryption and especially sending/receiving encrypted volumes. Another issue is that using swap on ZFS is still guaranteed to hang the kernel in low memory scenarios, because ZFS needs to allocate memory to write to swap.

The swap issue isn't zfs' fault though, it works just fine on FreeBSD and illumos... its a issue with how the Linux kernel handles things.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#42

Post by throwaway270925 »

harshreality wrote: post #27

Using DUP as the metadata profile sounds insane. Changing the metadata profile to at least raid1 (raid1, raid1c3, raid1c4) is a good idea, especially for anyone, against recommendations, using raid5 or raid6 for a btrfs array (raid1c3 is more appropriate for raid6). That would make it very difficult for metadata to get corrupted, which is the lion's share of the higher-impact problems with raid5/6 btrfs. check: btrfs…

This should be at the top, using metadata DUP on a 3 disk volume is already asking for it, and of course you loose data when you just use it as jbod with data stored only once. Unless this are enterprise disks with capacitors anything can happen when it suddenly looses power. Not the FSes fault.

With the same configuration this can happen with ZFS, bcachefs etc just as well.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#43

Post by throwaway270925 »

> A hard power cycle on a 3 device pool (data single, metadata DUP, DM-SMR disks) left the extent tree and free space tree in a state that no native repair path could resolve.

As a ZFS wrangler by day:

People in this thread seem to happily shit on btrfs here but this seems to be very much not like a sane, resilient configuration no matter the FS. Just something to keep in mind.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#44

Post by rcxdude »

throwaway270925 wrote: post #42
harshreality wrote: post #27

Using DUP as the metadata profile sounds insane. Changing the metadata profile to at least raid1 (raid1, raid1c3, raid1c4) is a good idea, especially for anyone, against recommendations, using raid5 or raid6 for a btrfs array (raid1c3 is more appropriate for raid6). That would make it very difficult for metadata to get corrupted, which is the lion's share of the higher-impact problems with raid5/6 btrfs. check: btrfs…

This should be at the top, using metadata DUP on a 3 disk volume is already asking for it, and of course you loose data when you just use it as jbod with data stored only once. Unless this are enterprise disks with capacitors anything can happen when it suddenly looses power. Not the FSes fault. With the same configuration this can happen with ZFS, bcachefs etc just as well.

Will it render the whole filesystem inaccessible and unrepairable on those filesystems as well? One of the issues with btrfs is that it's brittle: failure tends not to cause an inconsistency in the affected part of the filesystem but bring down the whole thing. In general people are a lot more understanding of a power failure resulting in data corruption around the files that are actively being written at the time (there are limits to how much consistency can be achieved here anyway), much less so when the blast radius expands a lot further.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#45

Post by Filligree »

duskdozer wrote: post #39

Welp. Guess I need to figure out another fs to use for a few drives in a nonraid pool I haven't gotten around to setting up yet. I forget why zfs seemed out. xfs?

ZFS is out because the Linux developers refuse to cooperate by providing the hooks it would need to avoid duplicating the disk cache.

That’s the only real reason. There are some papercuts, but they don’t compare to the risks described in this article.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#46

Post by Avamander »

jamesnorden wrote: post #30

People swear btrfs is "safe" now, but I've personally been bitten by data corruption more than once, so I stay away from it now.

I installed Fedora on BTRFS using their installer and I lost that partition entirely. Couldn't wrestle it back to life to even copy stuff off it.

I think what happened was that the machine ran out of battery in suspend, but an unclean shutdown shouldn't cause such a deep corruption.

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#47

Post by rincebrain »

Sesse__ wrote: post #36
fpoling wrote: post #18

Earlier quoted context omitted.

What are the alternatives to btrfs? At 12 TB data checksums are a must unless the data tolerate bit-rot. And if one wants to stick with the official kernel without out-of-tree modules, btrfs is the only choice.

Good thing all disks these days have data checksums, then! (50TB+ on ext4 and xfs, and no, no bit rot. Yes, I've checked most of it against separate sha256sum files now and then. As long as you have ECC RAM, disks just magically corrupting your data is largely a myth.)

Less mythic on SSDs than spinning rust, in my experience.

Not particularly frequent either way, but I have absolutely had models of SSDs where it became clear after a few months of use that a significant fraction of them appeared to be corrupting their internal state and serving incorrect data back to the host, leading to errors and panics.

(_usually_ this was accompanied by read or write errors. But _usually_ is notable when you've spent some time trying to figure out if the times it didn't were a different problem or the same problem but silent.)

There was also the notorious case with certain Samsung spinning rust and dropping data in their write cache if you issued SMART requests...

Re: Case study: recovery of a corrupted 12 TB multi-device pool

#49

Post by adrian_b »

rcxdude wrote: post #44
throwaway270925 wrote: post #42

Earlier quoted context omitted.

This should be at the top, using metadata DUP on a 3 disk volume is already asking for it, and of course you loose data when you just use it as jbod with data stored only once. Unless this are enterprise disks with capacitors anything can happen when it suddenly looses power. Not the FSes fault. With the same configuration this can happen with ZFS, bcachefs etc just as well.

Will it render the whole filesystem inaccessible and unrepairable on those filesystems as well? One of the issues with btrfs is that it's brittle: failure tends not to cause an inconsistency in the affected part of the filesystem but bring down the whole thing. In general people are a lot more understanding of a power failure resulting in data corruption around the files that are actively being written at the time (t…

A few decades ago, XFS was notorious because a power failure would wipe out various files, even if they had been opened only for reading. For instance, I had seen many systems that were bricked because XFS wiped out /etc/fstab after a power failure.

Nevertheless, many, many years ago, the XFS problems have been removed and today it is very robust.

During the last few years, I have seen a great number of power failures on some computers without a UPS, where XFS was used intensively at the moment of the power failure. Despite that, in none of those cases there was any filesystem corruption whatsoever, but the worst that has ever happened was the loss of the last writes performed immediately before the power failure.

This is the behavior that is expected from any file system that claims to be journaled, even in the past many journaled file systems failed to keep their promises, e.g. a few decades ago I had seen corrupted file systems on all existing Linux file systems and also on NTFS. At that time only the FreeBSD UFS with "soft updates" was completely unaffected by any kind of power failures.

However, nowadays I would expect all these file systems to be much more mature and to have fixed any bugs long ago.

BTRFS appears to be the exception, as the stories about corruption events do not seem to diminish in time.

Post reply on HN