Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Silent Data Corruption Is Real

changelog.complete.org

Post reply on HN
Print view View original

141–150 of 154 posts

Re: Silent Data Corruption Is Real

#141

Post by rleigh »

h2hn wrote: post #135
solarengineer wrote: post #131

Earlier quoted context omitted.

You can create a nested ZFS file system and set the number of copies of the various blocks to be two or more. This will take more space, but there'll be multiple copies of the same block of data. Ideally, though, please add an additional disk and set it up as a mirror. ZFS can detect the silent data corruption during data access or during a zpool scrub (which can be run on a live production server). If there happen t…

Got it but not for my use case then cause I don't want to halve my storage capacity. Anyway I will try to use it for my main PC which has several disks and continue to use my solution for single disk machines (laptop, vps, SoC...). :)

Note it won't necessarily halve the capacity. Selectively enable it for the datasets requiring it, and avoid the overhead with the rest.

Re: Silent Data Corruption Is Real

#142

Post by mdtancsa »

h2hn wrote: post #117
Laforet wrote: post #94

Earlier quoted context omitted.

FreeNAS != ZFS. The former is a specialised storage system that has to meet a very different set of criteria than a lightweight server with 1GB ram.

Is zfs able to repair from single data (copy) corruption? My main issue is to be able to repair a "silent" data corruption on a single drive machine. Am I able to use x% of my "partition" to data repair or do I need to use other partition/drive to mirror/raid it? If I understand right zfs can detect bitrot ("not really" a big deal) but without any local copy It can't self heal. My use case is an arm A20 SoC (lime2) t…

Not sure if it will fit your needs or not, but for long term storage on single HDs (and back in the day on DVD), I would create par files with about 5-10% redundancy to guard against data loss due to bad sectors. http://parchive.sourceforge.net/ total drive failure of course means loss of data, but the odd bad sector or corrupted bit would be correctable on a single disk. This was very popular back in the binary UseNet days....

Re: Silent Data Corruption Is Real

#143

Post by cmurf »

dom0 wrote: post #86
ak217 wrote: post #66

Earlier quoted context omitted.

crc32c is not weakish, and was chosen for a reason: crc32c has widespread hardware acceleration support that remains faster than any hash, and crc32c can be computed in parallel (unlike a hash, it has no hidden state, so you can sum independently computed block checksums to get the overall blob checksum). Bitrot detection doesn't need a cryptographic hash. You may want a hash for other purposes (like if you somehow t…

CRC32c -> I saw this many times fail to detect corruption on message lengths anywhere between a couple kB and a few MB. btrfs blocks are 16 kB iirc, so in range. The longer hashes of ZFS, Borg and so on mean that if it's corrupted I _definitely_ know. Not so confident with CRC32 from experience.

Not exactly, blocks are 4KiB so the vast majority of the CRC32C's apply to 4KiB block size; the metadata uses 16KiB nodes and those have their own checksum also. From what I see in the on disk format docs, it's a 20 byte checksum.

Re: Silent Data Corruption Is Real

#144

Post by cmurf »

JumpCrisscross wrote: post #100
Malic wrote: post #22

It's articles like this that re-enforce my disappointment that Apple is choosing to NOT implement checksums in their new file system, APFS. https://news.ycombinator.com/item?id=11934457

Can someone explain why one would checksum metadata but not user data? Is the assumption everything's backed up on iCloud? If so, are system files checksummed?

Metadata writes can be considered atomic, so an integrated checksum is written at the time the metadata block is written, or overwritten. Whereas with data, you can't do overwrites of either data or checksum, it's not atomic and any kind of crash or powerfailure will result in mismatching data to checksum. So unless you have something really clever to work around this, you need a copy on write file system to do data checksums.

Re: Silent Data Corruption Is Real

#145

Post by cmurf »

blablabloe wrote: post #124

The story here is not how Silent Data Corruption is real. The story is that somebody did a bad home brew server build and fucked up. So ZFS protects against end-user mistakes. I was really hoping about a story on some large-scale study on silent data corruption, but no, just an ankedote. Sad! :D

What was the flaw in the setup that makes this not silent data corruption? That any other file system would not have caught the problem means corruption would have silently propagated to the application layer. And yet it didn't, because the corruption was detected. I fail to understand the point of this comment.

Re: Silent Data Corruption Is Real

#146

Post by bestham »

mrb wrote: post #12

The exact same silent data corruption issues just happened to my 6 x 5TB ZFS FreeBSD fileserver. But unlike what the poster concluded, mine were caused by bad (ECC!) RAM. I kept meticulous notes, so here is my story... I scrub on a weekly basis. One day ZFS started reporting silent errors on disk ada3, just 4kB: pool: tank state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt wa…

Is it not "5 choose 4" since one of the 6 is a "known bad" in terms of ZFS knowledge?

Re: Silent Data Corruption Is Real

#147

Post by bestham »

rleigh wrote: post #141
h2hn wrote: post #135

Earlier quoted context omitted.

Got it but not for my use case then cause I don't want to halve my storage capacity. Anyway I will try to use it for my main PC which has several disks and continue to use my solution for single disk machines (laptop, vps, SoC...). :)

Note it won't necessarily halve the capacity. Selectively enable it for the datasets requiring it, and avoid the overhead with the rest.

No but parity archives solves a different problem, with only some percent of wasted storage you can survive bit-errors in your dataset. It's like reed-solomon for files.

In order to achive the same with ZFS you have to run RAID-Z2 on sparse files.

Re: Silent Data Corruption Is Real

#148

Post by jepler »

mrb wrote: post #138
jepler wrote: post #120

Earlier quoted context omitted.

Specifically, I wrote a program to search for single-bit-flip collisions in sha1 truncated to 16 bits. The program didn't need to search for long before finding two messages with the same 16-truncated sha1 with a single bit flip at bit 1 of byte 171 of a 256-byte message. 376 1 171 be44b935e7ecfc81d1fe2cddcd7c1d7e04338fd83fa994cd6a877732ca5d8db83346bd9ccbfc4c8770682bd307c782421a512a80a106be87825d5c13f3156e23ffaacdfc1…

You are not testing a crypto hash. "Crypto hash" means it is cryptographically strong, not truncated to 16 bits. For example ZFS with checksum=sha256 will use the full 256-bit hash for detecting data corruption.

Yup, you're right. if you use a full size cryptographic hash then the number of undetected errors can be treated as 0 regardless of hamming distance. On the other hand, it has 8x the storage overhead of a 32-bit CRC.

Re: Silent Data Corruption Is Real

#149

Post by mrb »

bestham wrote: post #146
mrb wrote: post #12

The exact same silent data corruption issues just happened to my 6 x 5TB ZFS FreeBSD fileserver. But unlike what the poster concluded, mine were caused by bad (ECC!) RAM. I kept meticulous notes, so here is my story... I scrub on a weekly basis. One day ZFS started reporting silent errors on disk ada3, just 4kB: pool: tank state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt wa…

Is it not "5 choose 4" since one of the 6 is a "known bad" in terms of ZFS knowledge?

ZFS does not know which disk(s) caused the silent data corruption. It first assumes only 1 disk is bad and makes 4 attempts to rebuild it. It assumes successively each one of the 4 data stripes is bad, and rebuilds it from the parity stripes. But if it still cannot rebuild the correct data it means 2 disks are bad, so it tries 15 combinations of data/parity stripes to rebuild the 4 data stripes (well... minus the combination of "4 data stripes" which is already known bad, so the 14 other combinations are tested).

Re: Silent Data Corruption Is Real

#150

Post by mrb »

blablabloe wrote: post #127
mrb wrote: post #12

The exact same silent data corruption issues just happened to my 6 x 5TB ZFS FreeBSD fileserver. But unlike what the poster concluded, mine were caused by bad (ECC!) RAM. I kept meticulous notes, so here is my story... I scrub on a weekly basis. One day ZFS started reporting silent errors on disk ada3, just 4kB: pool: tank state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt wa…

When a double-parity error is detected, the operating system should halt. Maybe that didn't happen properly. Tripple parity errors may go undetected, but how likely is that. I wonder what 'really' happened.

I saw dozens of ECC errors in the IPMI log, and dozens of MCAs in dmesg. So the memory was failing so badly that it is probably there were 3+ bit errors.
Post reply on HN