Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Silent Data Corruption Is Real

changelog.complete.org

Post reply on HN
Print view View original

131–140 of 154 posts

Re: Silent Data Corruption Is Real

#131

Post by solarengineer »

h2hn wrote: post #117
Laforet wrote: post #94

Earlier quoted context omitted.

FreeNAS != ZFS. The former is a specialised storage system that has to meet a very different set of criteria than a lightweight server with 1GB ram.

Is zfs able to repair from single data (copy) corruption? My main issue is to be able to repair a "silent" data corruption on a single drive machine. Am I able to use x% of my "partition" to data repair or do I need to use other partition/drive to mirror/raid it? If I understand right zfs can detect bitrot ("not really" a big deal) but without any local copy It can't self heal. My use case is an arm A20 SoC (lime2) t…

You can create a nested ZFS file system and set the number of copies of the various blocks to be two or more. This will take more space, but there'll be multiple copies of the same block of data.

Ideally, though, please add an additional disk and set it up as a mirror.

ZFS can detect the silent data corruption during data access or during a zpool scrub (which can be run on a live production server). If there happen to be multiple copies, then ZFS can use one of the working copies to repair the corrupted copy.

Re: Silent Data Corruption Is Real

#132

Post by fnord123 »

pmarreck wrote: post #126
fnord123 wrote: post #111

Earlier quoted context omitted.

Sorry, a better answer is that it's already done in the hardware/firmware layer. e.g. from Wikipedia[1]: """ Modern HDDs present a consistent interface to the rest of the computer, no matter what data encoding scheme is used internally. Typically a DSP in the electronics inside the HDD takes the raw analog voltages from the read head and uses PRML and Reed–Solomon error correction[144] to decode the sector boundaries…

If this is the case, then why does bit-rot still occur?

It's not magic. If you have enough bit flips for the same data then it's not recoverable. Sometimes it could even flip and resemble correct data. It might not happen to you, but if 500 people reading this thread each have access to 50T of data, then sure bit rot can happen in some of that 25 petabytes.

Re: Silent Data Corruption Is Real

#133

Post by jgoerzen »

blablabloe wrote: post #124

The story here is not how Silent Data Corruption is real. The story is that somebody did a bad home brew server build and fucked up. So ZFS protects against end-user mistakes. I was really hoping about a story on some large-scale study on silent data corruption, but no, just an ankedote. Sad! :D

I didn't go into detail in the article, but the server in question is running a Supermicro X10SLH-F-O motherboard, ECC RAM, and a Haswell CPU, in a Rosewill RSV-L4411 4U chassis. Is there a hardware problem here? For sure. But you can't write this off as being some dusty overclock mess bought at someone's garage sale.

I have, incidentally, seen this in corporate environments on traditionally-engineered sever-class hardware as well. This is just a much more easily-discussed case.

Re: Silent Data Corruption Is Real

#134

Post by zzzcpan »

gbrown_ wrote: post #73
zzzcpan wrote: post #34

Earlier quoted context omitted.

That's the thing, identifying the problem on the filesystem level is useless if you can only correct it on another level. Unless your filesystem is distributed and self-healing, it's not a place for checksums, it must remain a thin predictable layer on top of a disk.

> That's the thing, identifying the problem on the filesystem level is useless if you can only correct it on another level. Nope I'd take a system that can tell me "Hey sorry I don't have the right data" over "Here's some data, looks good to me" any day of the week. Also as ProblemFactory points out, ZFS will self-heal whenever possible.

"Nope I'd take a system that can tell me "Hey sorry I don't have the right data" over "Here's some data, looks good to me" any day of the week."

Nope, it doesn't help with anything. I think you are making false assumptions about the safety of your data. Because to keep consistency such system must automatically shutdown on data corruption and only wake up once the problem is fixed. To keep availability it must automatically fix the problem. There is no third option, because filesystems are not aware of how applications use them. Returning EIO would also introduce inconsistency.

But if you actually care about your data, you shouldn't trust a single box, ever, no matter how fancy the filesystem is. There are just so many things that could go wrong, it's hard to believe someone would even take that risk.

Just a few weeks ago I saw data loss on one of the object storage nodes, because of a typical sata cable issue. It caused multiple kernel panics, affecting data on multiple disks, before I was able to figure it out. Not a problem for a distributed storage, but you wouldn't survive that with a local storage. This is also one of the reasons I consider zfs an over engineered toy, that doesn't target real world problems.

Re: Silent Data Corruption Is Real

#135

Post by h2hn »

solarengineer wrote: post #131
h2hn wrote: post #117

Earlier quoted context omitted.

Is zfs able to repair from single data (copy) corruption? My main issue is to be able to repair a "silent" data corruption on a single drive machine. Am I able to use x% of my "partition" to data repair or do I need to use other partition/drive to mirror/raid it? If I understand right zfs can detect bitrot ("not really" a big deal) but without any local copy It can't self heal. My use case is an arm A20 SoC (lime2) t…

You can create a nested ZFS file system and set the number of copies of the various blocks to be two or more. This will take more space, but there'll be multiple copies of the same block of data. Ideally, though, please add an additional disk and set it up as a mirror. ZFS can detect the silent data corruption during data access or during a zpool scrub (which can be run on a live production server). If there happen t…

Got it but not for my use case then cause I don't want to halve my storage capacity.

Anyway I will try to use it for my main PC which has several disks and continue to use my solution for single disk machines (laptop, vps, SoC...). :)

Re: Silent Data Corruption Is Real

#136

Post by mrb »

h2hn wrote: post #121
mrb wrote: post #98

Earlier quoted context omitted.

It depends. Metadata is always redundant with 2 copies (even when using copies=1). So if the file's metadata is corrupted, yes ZFS will fully recover and rewrite a 2nd good copy of the metadata. But if the data is corrupted, then ZFS can do nothing to recover (you may be able to partially read the file, but the rest of it will return I/O errors.)

Ok, so zfs for single drive users doesn't fix single data corruption. Definitely I'm going to use my solution so. All the next generation FS stuff is cool (btrfs also indeed) but for the simplest use case people just need safe data and fix it it the disk goes bad.

Why don't you use copies=2 with a single disk?

Re: Silent Data Corruption Is Real

#137

Post by marcosdumay »

ak217 wrote: post #89
dom0 wrote: post #86

Earlier quoted context omitted.

CRC32c -> I saw this many times fail to detect corruption on message lengths anywhere between a couple kB and a few MB. btrfs blocks are 16 kB iirc, so in range. The longer hashes of ZFS, Borg and so on mean that if it's corrupted I _definitely_ know. Not so confident with CRC32 from experience.

I'm curious about the setting in which you saw these failures, could you elaborate? Unlike a plain checksum, CRC-32C is hardened against bias, which means its distribution is not far from that of an ideal checksum. This means if your bitrot is random and you're using 16KB blocks, you will need to see on the order of ((2 * * 32) * 16KB)=64TB of corrupted data to get a random failure. Modern hard drives corrupt data at…

For disk storage CRC-32C is still non-broken. You can't say the same about on-board communication protocols or even some LANs.

When people started using CRC-32 it was because with the technology of the time it was virtually impossible to see collisions. Nowadays we are discussing if it's reasonable to expect a data volume that gives you 40% or 60% of collision chance.

CRC32 end is way overdue. We should standardize on a CRC64 algorithm soon, or will have our hands forced and probably stick with a bad choice.

Re: Silent Data Corruption Is Real

#138

Post by mrb »

jepler wrote: post #120
mrb wrote: post #102

Earlier quoted context omitted.

" Not only is a cryptographic hash unnecessary, under certain circumstances it will actually do a worse job. " A cryptographic hash is unnecessary, but as you point out in the next to last paragraph, it is statistically improbable that it will do a worse job. Because collisions are statistically improbable.

Specifically, I wrote a program to search for single-bit-flip collisions in sha1 truncated to 16 bits. The program didn't need to search for long before finding two messages with the same 16-truncated sha1 with a single bit flip at bit 1 of byte 171 of a 256-byte message. 376 1 171 be44b935e7ecfc81d1fe2cddcd7c1d7e04338fd83fa994cd6a877732ca5d8db83346bd9ccbfc4c8770682bd307c782421a512a80a106be87825d5c13f3156e23ffaacdfc1…

You are not testing a crypto hash. "Crypto hash" means it is cryptographically strong, not truncated to 16 bits. For example ZFS with checksum=sha256 will use the full 256-bit hash for detecting data corruption.

Re: Silent Data Corruption Is Real

#139

Post by pmarreck »

fnord123 wrote: post #132
pmarreck wrote: post #126

Earlier quoted context omitted.

If this is the case, then why does bit-rot still occur?

It's not magic. If you have enough bit flips for the same data then it's not recoverable. Sometimes it could even flip and resemble correct data. It might not happen to you, but if 500 people reading this thread each have access to 50T of data, then sure bit rot can happen in some of that 25 petabytes.

Do you think that there is a need for some kind of ECC at the OS level? There are surely some applications where even 1 bit flip in 25 petabytes is bad. Amazon's 2008 site outage comes to mind: http://status.aws.amazon.com/s3-20080720.html

Re: Silent Data Corruption Is Real

#140

Post by fnord123 »

pmarreck wrote: post #139
fnord123 wrote: post #132

Earlier quoted context omitted.

It's not magic. If you have enough bit flips for the same data then it's not recoverable. Sometimes it could even flip and resemble correct data. It might not happen to you, but if 500 people reading this thread each have access to 50T of data, then sure bit rot can happen in some of that 25 petabytes.

Do you think that there is a need for some kind of ECC at the OS level? There are surely some applications where even 1 bit flip in 25 petabytes is bad. Amazon's 2008 site outage comes to mind: http://status.aws.amazon.com/s3-20080720.html

Yes, that's why you have erasure coding across drives either using RAID groups or using an object store. Just not at the single drive file system level where blocks going bad within a disk tend to be correlated (i.e. the drive is getting old and will die).

Also, EC at the block level would probably spread the blocks around the drive. This means any read would need to seek all over the dang place trying to reassemble files and that's a bad access pattern for spinning disks. Real bad. Like, the worst. It might even reduce the effective lifetime of the drive. So it would be not only correlated with device failure, it could precipitate it.

Maybe it would be ok on an SSD.

Post reply on HN