Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Silent Data Corruption Is Real

changelog.complete.org

Post reply on HN
Print view View original

21–30 of 154 posts

Re: Silent Data Corruption Is Real

#21

Post by djsumdog »

seiferteric wrote: post #15
nayuki wrote: post #6

Earlier quoted context omitted.

Btrfs has experienced some data loss bugs in recent memory. It looks like ZFS is the only remaining option. https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Da... https://www.spinics.net/lists/linux-btrfs/msg59190.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...

Yes I have tried to use btrfs several times for work projects and personally because I was very excited about it but every time I have ran into severe bugs even though it was said to be "stable". I have given up for now, maybe I will check back in a couple more years.

The last time I was using btrfs was around 2014, and I was wondering why my hard drive was always showing 100% utilization even after I moved/deleted a ton-o-stuff.

Turns out at the time, re-balancing still had to be run manually. I'm not sure if that still holds true.

Re: Silent Data Corruption Is Real

#23

Post by dom0 »

elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

btrfs only uses CRC32c which is weakish. ZFS is great but not exactly portable. I started to use Borg now for archiving purposes as well, not just backup. For me (low access concurrency, i.e. single or at most "a few" users) that works very well. Portable + strong checksumming + strong crypto + mountable + reasonable speed (with prospect of more) is a good package. It doesn't solve error correction, though.

Re: Silent Data Corruption Is Real

#24

Post by X86BSD »

elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

If you actually care about your data you simply will have to switch platforms. Illumos or FreeBSD. Linux has no answer to the horrors of garbage Filesystems and garbage lying hardware, firmware etc that has plagued UNIX admins for ever. Btrfs is apparently the best they can do and its glaringly deficient architecturally.

Filesystems are HARD. And as bcantrill notes, if you can't get simple things like epoll right your dead in the water on harder things like kqueue and zfs, Dtrace, jails, etc.

This is proving true looking at the fact Linux land can't get a filesystem right and they've been trying for decades.

Many will argue and try and justify that simple fact but as the cold harsh hammer of reality slams into their skull eventually those folks will be forced to face the reality.

Linux honestly just needs to adopt zfs. Period. But because of the Linux license that may be all but impossible now. Which leaves Linux in this untenable position of being stuck in a tar pit. Unable to adopt the clear choice to move forward and unable to have the technical ability to implement their own solution equal to zfs. So wither now Linux?

Re: Silent Data Corruption Is Real

#25

Post by heinrich5991 »

meesterdude wrote: post #4

interesting find! I wonder what would be a good safeguard to this. I feel like just backing up your data would offer something - but a file could silently change and become corrupted in the backup too.

You could use ZFS, then the file cannot silently become corrupted.

Re: Silent Data Corruption Is Real

#26

Post by [deleted] »

elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

[deleted]

Re: Silent Data Corruption Is Real

#27

Post by tombrossman »

nayuki wrote: post #6
elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

Btrfs has experienced some data loss bugs in recent memory. It looks like ZFS is the only remaining option. https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Da... https://www.spinics.net/lists/linux-btrfs/msg59190.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...

Any idea if ZFS plays well with Ubuntu's full-disk encryption? I've used the FDE option at install for years and every time I upgrade (I wipe & reinstall every year or so) I try to understand how to first set up ZFS, then FDE, and then I realize it's far too complicated for me. Any good tutorials or setup guides that even a moron could understand?

I've got a pretty good setup now with a fairly complex fstab, multiple SSDs, backup drives, and everything fully encrypted and auto mounting at boot. I'd really love to move this to a file system more resistant to data corruption.

Re: Silent Data Corruption Is Real

#28

Post by X86BSD »

dom0 wrote: post #23
elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

btrfs only uses CRC32c which is weakish. ZFS is great but not exactly portable. I started to use Borg now for archiving purposes as well, not just backup. For me (low access concurrency, i.e. single or at most "a few" users) that works very well. Portable + strong checksumming + strong crypto + mountable + reasonable speed (with prospect of more) is a good package. It doesn't solve error correction, though.

ZFS not portable? Have you ever used it?

"ZFS export" on the host system, remove drives, insert drives in new server, "ZFS import" on the new server, It really is that simple.

Re: Silent Data Corruption Is Real

#29

Post by elfchief »

zzzcpan wrote: post #13
elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

What would you do with a failed checksum on a filesystem level? These errors could be and likely to be transient too.

If nothing else, you can log the error. If you have RAID1 you can recopy the block from a good copy. It is, honestly, probably situation-specific, but step #1 is always going to be "identify that you have a problem"

Re: Silent Data Corruption Is Real

#30

Post by icebraining »

ralfd wrote: post #7
meesterdude wrote: post #4

interesting find! I wonder what would be a good safeguard to this. I feel like just backing up your data would offer something - but a file could silently change and become corrupted in the backup too.

Hm. You could make two backups and checksum each file and safe the checksum. Then you could compare regularly file contents with the initial checksum, if there is a mismatch copy from the other backup.

Git-annex is nice for this; the fsck command will check the file against a checksum and request a new copy from other node automatically if the check fails.
Post reply on HN