Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Silent Data Corruption Is Real

changelog.complete.org

Post reply on HN
Print view View original

31–40 of 154 posts

Re: Silent Data Corruption Is Real

#31

Post by elfchief »

morecoffee wrote: post #19
elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

Making data on individual drives is probably not a long term goal for integrity. It is more likely that multi homing data in geographically disjoint locations with some sort of syncing is a better long term goal. Fixing silent data corruption on a single drive doesn't solve any of the much more likely disasters, like fire, flooding, weather, etc. Not even datacenters can withstand lightning.

The problem with multiple geographic locations, in this context, is that you'd have to read from all of them, and compare the results, to know that you have file corruption. Which is, needless to say, not something that it makes sense to do.

The purpose of data block checksumming isn't to make your data more resilient (at least directly), it's to make sure you know you have a problem. Once you know you have a problem, then you can go read from your alternate datacenter or whatever.

Re: Silent Data Corruption Is Real

#32

Post by dredmorbius »

"Data tends to corrupt. Absolute data tends to corrupt absolutely."

In both sense of the word.

Many moons ago, in one of my first professional assignments, I was tasked with what was, for the organisation, myself, and the provisioned equipment, a stupidly large data processing task. One of the problems encountered was a failure of a critical hard drive -- this on a system with no concept of a filesystem integrity check (think a particularly culpable damned operating system, and yes, I said that everything about this was stupid). The process of both tracking down, and then demonstrating convincingly to management (I said ...) the nature of the problem was infuriating.

And that was with hardware which was reliably and replicably bad. Transient data corruption ... because cosmic rays ... gets to be one of those particularly annoying failure modes.

Yes, checksums and redundancy, please.

Re: Silent Data Corruption Is Real

#33

Post by stock_toaster »

X86BSD wrote: post #24
elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

If you actually care about your data you simply will have to switch platforms. Illumos or FreeBSD. Linux has no answer to the horrors of garbage Filesystems and garbage lying hardware, firmware etc that has plagued UNIX admins for ever. Btrfs is apparently the best they can do and its glaringly deficient architecturally. Filesystems are HARD. And as bcantrill notes, if you can't get simple things like epoll right you…

  > Linux honestly just needs to adopt zfs. Period.
  > But because of the Linux license that may be all but impossible now.
Hammer2 might be an option, once it gets finished.

Re: Silent Data Corruption Is Real

#34

Post by zzzcpan »

elfchief wrote: post #29
zzzcpan wrote: post #13

Earlier quoted context omitted.

What would you do with a failed checksum on a filesystem level? These errors could be and likely to be transient too.

If nothing else, you can log the error. If you have RAID1 you can recopy the block from a good copy. It is, honestly, probably situation-specific, but step #1 is always going to be "identify that you have a problem"

That's the thing, identifying the problem on the filesystem level is useless if you can only correct it on another level. Unless your filesystem is distributed and self-healing, it's not a place for checksums, it must remain a thin predictable layer on top of a disk.

Re: Silent Data Corruption Is Real

#35

Post by heinrich5991 »

zzzcpan wrote: post #34
elfchief wrote: post #29

Earlier quoted context omitted.

If nothing else, you can log the error. If you have RAID1 you can recopy the block from a good copy. It is, honestly, probably situation-specific, but step #1 is always going to be "identify that you have a problem"

That's the thing, identifying the problem on the filesystem level is useless if you can only correct it on another level. Unless your filesystem is distributed and self-healing, it's not a place for checksums, it must remain a thin predictable layer on top of a disk.

What is wrong with returning an IO error (e.g. EIO) if the data read from the disk is corrupt?

Re: Silent Data Corruption Is Real

#36

Post by adrianN »

zzzcpan wrote: post #13
elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

What would you do with a failed checksum on a filesystem level? These errors could be and likely to be transient too.

Use an error correcting code, fix errors you can fix, report unrecoverable errors to the user so that they can restore from backup.

Re: Silent Data Corruption Is Real

#37

Post by dpedu »

X86BSD wrote: post #28
dom0 wrote: post #23

Earlier quoted context omitted.

btrfs only uses CRC32c which is weakish. ZFS is great but not exactly portable. I started to use Borg now for archiving purposes as well, not just backup. For me (low access concurrency, i.e. single or at most "a few" users) that works very well. Portable + strong checksumming + strong crypto + mountable + reasonable speed (with prospect of more) is a good package. It doesn't solve error correction, though.

ZFS not portable? Have you ever used it? "ZFS export" on the host system, remove drives, insert drives in new server, "ZFS import" on the new server, It really is that simple.

I think he means portability of the software. What I want to 'ZFS import' on a Windows or macOS machine?

Re: Silent Data Corruption Is Real

#38

Post by dpedu »

heinrich5991 wrote: post #25
meesterdude wrote: post #4

interesting find! I wonder what would be a good safeguard to this. I feel like just backing up your data would offer something - but a file could silently change and become corrupted in the backup too.

You could use ZFS, then the file cannot silently become corrupted.

Yes it can. ZFS will only notice next time you scrub or read the sector.

Re: Silent Data Corruption Is Real

#39

Post by Veratyr »

elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

bcachefs is working on it but needs support:

- https://bcache.evilpiepirate.org/Bcachefs/

- https://www.patreon.com/bcachefs/

(I'm not Kent but I love the work)

Re: Silent Data Corruption Is Real

#40

Post by justincormack »

dpedu wrote: post #37
X86BSD wrote: post #28

Earlier quoted context omitted.

ZFS not portable? Have you ever used it? "ZFS export" on the host system, remove drives, insert drives in new server, "ZFS import" on the new server, It really is that simple.

I think he means portability of the software. What I want to 'ZFS import' on a Windows or macOS machine?

ZFS on OSX has been revived I believe.
Post reply on HN