Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Silent Data Corruption Is Real

changelog.complete.org

Post reply on HN
Print view View original

91–100 of 154 posts

Re: Silent Data Corruption Is Real

#91

Post by delroth »

djsumdog wrote: post #21
seiferteric wrote: post #15

Earlier quoted context omitted.

Yes I have tried to use btrfs several times for work projects and personally because I was very excited about it but every time I have ran into severe bugs even though it was said to be "stable". I have given up for now, maybe I will check back in a couple more years.

The last time I was using btrfs was around 2014, and I was wondering why my hard drive was always showing 100% utilization even after I moved/deleted a ton-o-stuff. Turns out at the time, re-balancing still had to be run manually. I'm not sure if that still holds true.

It is still a problem, I just had one of my personal servers hit issues where I couldn't even manually re-balance because the metadata was full. Had to apply weird workarounds to be able to write to that filesystem again...

Re: Silent Data Corruption Is Real

#92

Post by cyphar »

elfchief wrote: post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

> (...sure, you could call zfs or btrfs "mainstream", I suppose, but when I say "mainstream" I mean something along the lines of "officially supported by RedHat". zfs isn't, and RH considers btrfs to still be "experimental".)

SUSE (the other main enterprise Linux distribution vendor) has been supporting Btrfs in the enterprise (as the default filesystem) for several years.

[Disclosure: I work for SUSE.]

Re: Silent Data Corruption Is Real

#93

Post by bascule »

ak217 wrote: post #66
dom0 wrote: post #23

Earlier quoted context omitted.

btrfs only uses CRC32c which is weakish. ZFS is great but not exactly portable. I started to use Borg now for archiving purposes as well, not just backup. For me (low access concurrency, i.e. single or at most "a few" users) that works very well. Portable + strong checksumming + strong crypto + mountable + reasonable speed (with prospect of more) is a good package. It doesn't solve error correction, though.

crc32c is not weakish, and was chosen for a reason: crc32c has widespread hardware acceleration support that remains faster than any hash, and crc32c can be computed in parallel (unlike a hash, it has no hidden state, so you can sum independently computed block checksums to get the overall blob checksum). Bitrot detection doesn't need a cryptographic hash. You may want a hash for other purposes (like if you somehow t…

Bitrot detection doesn't need a cryptographic hash.

Not only is a cryptographic hash unnecessary, under certain circumstances it will actually do a worse job.

Cryptographic hashes operate under a different set of constraints than error detecting code. With an error detecting code, it's desirable to guarantee a different checksum in the event of a bitflip.

With a cryptographic random oracle, this is not the case: we want all outcomes to have equal probability, even potentially producing the same digest in the event of a bitflip. As an example of a system which failed in this way: Enigma was specifically designed so the ciphertext of a given letter was always different from its plaintext. This left a statistical signature on the ciphertext which in turn was exploited by cryptanalysts to recover plaintexts. (Note: a comparison to block ciphers is apt as most hash functions are, at their core, similar to block ciphers)

Though the digests of cryptographic hash functions are so large it's statistically improbable for a single bitflip to result in the same digest as the original, it is not a guarantee the same way it is with the CRC family.

Cryptographic hash functions are not designed to be error detecting codes. They are designed to be random oracles. Outside a security context, using a CRC family function will not only be faster, but will actually provide guarantees cryptographic hash functions can't.

Re: Silent Data Corruption Is Real

#94

Post by Laforet »

h2hn wrote: post #71
tnorgaard wrote: post #63

Earlier quoted context omitted.

Please stop spreading this misinformed statement. I assume you are referring to the ZFS ARC (Adaptive Replacement Cache). It works in much the same way as a regular Linux page cache. It does not take much more memory (if you disable prefetch) and will only use what is available/idle. We use Linux with ZFS on production systems with as low as 1GB memory. We stopped counting the times it has saved the day. :-) ECC is a…

Interesting. That's rigth the kind of hardware I was referring to, 1 GB of plain RAM. Truly, I haven't tested ZFS yet for that reason I've always read that ZFS has big requirements so I refrained to try it. It seems I should give it a try. ;) Btrfs is another story I've used it for years and I'd prefer not to have to use it anymore untill it'll become "stable" and "performance". :)

FreeNAS != ZFS. The former is a specialised storage system that has to meet a very different set of criteria than a lightweight server with 1GB ram.

Re: Silent Data Corruption Is Real

#95

Post by speleo_engr »

ak217 wrote: post #89
dom0 wrote: post #86

Earlier quoted context omitted.

CRC32c -> I saw this many times fail to detect corruption on message lengths anywhere between a couple kB and a few MB. btrfs blocks are 16 kB iirc, so in range. The longer hashes of ZFS, Borg and so on mean that if it's corrupted I _definitely_ know. Not so confident with CRC32 from experience.

I'm curious about the setting in which you saw these failures, could you elaborate? Unlike a plain checksum, CRC-32C is hardened against bias, which means its distribution is not far from that of an ideal checksum. This means if your bitrot is random and you're using 16KB blocks, you will need to see on the order of ((2 * * 32) * 16KB)=64TB of corrupted data to get a random failure. Modern hard drives corrupt data at…

Posts like this are what keep me coming back to HN.

Re: Silent Data Corruption Is Real

#96

Post by 3legcat »

tnorgaard wrote: post #63
h2hn wrote: post #59

I started a really simple and effective project the last month to be able to fix from bitrot in linux(MacOs/Unix?). It's "almost done" just need more real testing and make the systemd service. I've been pretty busy the last weeks so I've only been able to improve the bitrot performance. https://github.com/liloman/heal-bitrots Unfortunatly, btrfs is not stable and zfs needs a "super computer" or at least as much GBs o…

Please stop spreading this misinformed statement. I assume you are referring to the ZFS ARC (Adaptive Replacement Cache). It works in much the same way as a regular Linux page cache. It does not take much more memory (if you disable prefetch) and will only use what is available/idle. We use Linux with ZFS on production systems with as low as 1GB memory. We stopped counting the times it has saved the day. :-) ECC is a…

> ECC is nice to have.

Actually it seems ECC is important for ZFS filesystems see:

http://louwrentius.com/please-use-zfs-with-ecc-memory.html

Re: Silent Data Corruption Is Real

#97

Post by rsync »

Malic wrote: post #22

It's articles like this that re-enforce my disappointment that Apple is choosing to NOT implement checksums in their new file system, APFS. https://news.ycombinator.com/item?id=11934457

iDevices don't have multi-TB disks, nor do they have arrays of disks.

Therefore this is of limited relevance to Apple.

Re: Silent Data Corruption Is Real

#98

Post by mrb »

swinglock wrote: post #83

If I were to run ZFS on my laptop with a single disk and copies=1, and a file becomes corrupted, can I recover it (partially)? My assumption is the read will fail and the error logged but there is no redundancy so it will stay unreadable. Will ZFS attempt to read the file again, in case the error is transient? If not, can I make ZFS retry reading? Can I "unlock" the file and read it even though it is corrupted, or ge…

It depends. Metadata is always redundant with 2 copies (even when using copies=1). So if the file's metadata is corrupted, yes ZFS will fully recover and rewrite a 2nd good copy of the metadata. But if the data is corrupted, then ZFS can do nothing to recover (you may be able to partially read the file, but the rest of it will return I/O errors.)

Re: Silent Data Corruption Is Real

#99

Post by problems »

Dylan16807 wrote: post #74
gjjrfcbugxbhf wrote: post #65

Earlier quoted context omitted.

Would a FUSE implementation of ZFS not be possible? (Just wondering I've no idea what's possible here)

A FUSE implementation of ZFS exists and works well, and adding FUSE support to the Windows 10 Linux subsystem appears to be reasonably high up on the priority list. That doesn't get you access from Windows programs, but there are some other ways to do FUSE or FUSE-like things on Windows..

It may be fairly straightforward to port the ZFS FUSE to Windows if you use things like Dokan or WinFsp which have the FUSE interface supported fairly well - these would give full access via standard Windows tools.

Re: Silent Data Corruption Is Real

#100

Post by JumpCrisscross »

Malic wrote: post #22

It's articles like this that re-enforce my disappointment that Apple is choosing to NOT implement checksums in their new file system, APFS. https://news.ycombinator.com/item?id=11934457

Can someone explain why one would checksum metadata but not user data? Is the assumption everything's backed up on iCloud? If so, are system files checksummed?
Post reply on HN