Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Silent Data Corruption Is Real

changelog.complete.org

Post reply on HN
Print view View original

121–130 of 154 posts

Re: Silent Data Corruption Is Real

#121

Post by h2hn »

mrb wrote: post #98
swinglock wrote: post #83

If I were to run ZFS on my laptop with a single disk and copies=1, and a file becomes corrupted, can I recover it (partially)? My assumption is the read will fail and the error logged but there is no redundancy so it will stay unreadable. Will ZFS attempt to read the file again, in case the error is transient? If not, can I make ZFS retry reading? Can I "unlock" the file and read it even though it is corrupted, or ge…

It depends. Metadata is always redundant with 2 copies (even when using copies=1). So if the file's metadata is corrupted, yes ZFS will fully recover and rewrite a 2nd good copy of the metadata. But if the data is corrupted, then ZFS can do nothing to recover (you may be able to partially read the file, but the rest of it will return I/O errors.)

Ok, so zfs for single drive users doesn't fix single data corruption.

Definitely I'm going to use my solution so. All the next generation FS stuff is cool (btrfs also indeed) but for the simplest use case people just need safe data and fix it it the disk goes bad.

Re: Silent Data Corruption Is Real

#122

Post by [deleted] »

vardump wrote: post #115
ak217 wrote: post #89

Earlier quoted context omitted.

I'm curious about the setting in which you saw these failures, could you elaborate? Unlike a plain checksum, CRC-32C is hardened against bias, which means its distribution is not far from that of an ideal checksum. This means if your bitrot is random and you're using 16KB blocks, you will need to see on the order of ((2 * * 32) * 16KB)=64TB of corrupted data to get a random failure. Modern hard drives corrupt data at…

> This means if your bitrot is random and you're using 16KB blocks, you will need to see on the order of ((2 * * 32) * 16KB)=64TB of corrupted data to get a random failure. No, that's what you need to generate one guaranteed failure, when enumerating different random corruption possibilities. Simply because 32-bit number can at most represent 2^32 different states. In practice, you'd have 50% probability to have a co…

[deleted]

Re: Silent Data Corruption Is Real

#123

Post by pmarreck »

Of course, my ZFS NAS backup is sound until a file that got bitrotted on my non-ZFS computer is touched and then backed up to it :/

It's kind of (literally?) like immutability. If you allow even a little mutability, it ruins it.

I think all filesystems should be able to add error-correction data to ensure data integrity.

Re: Silent Data Corruption Is Real

#124

Post by blablabloe »

The story here is not how Silent Data Corruption is real. The story is that somebody did a bad home brew server build and fucked up.

So ZFS protects against end-user mistakes.

I was really hoping about a story on some large-scale study on silent data corruption, but no, just an ankedote.

Sad!

:D

Re: Silent Data Corruption Is Real

#125

Post by blablabloe »

kabdib wrote: post #17

Oh, yes. Silent bit errors are tons of fun to track down. I spent a day chasing what turned out to be a bad bit in the cache of a disk drive; bits would get set to zero in random sectors, but always at a specific sector offset. The drive firmware didn't bother doing any kind of memory test; even a simple stuck-at test would have found this and preserved the customer's data. In another case, we had Merkle-tree integri…

Enterprise disks do have ECC cache as opposed to consumer drives. Was it a consumer drive?

Re: Silent Data Corruption Is Real

#126

Post by pmarreck »

fnord123 wrote: post #111
fnord123 wrote: post #85

Earlier quoted context omitted.

Sector failures on a single drive are not independent random events. So doing this on a single drive is not perceived to be a good idea. The only choices, imo, are multiple disks, or automatic backup to offsite (dropbox, box, one drive, etc).

Sorry, a better answer is that it's already done in the hardware/firmware layer. e.g. from Wikipedia[1]: """ Modern HDDs present a consistent interface to the rest of the computer, no matter what data encoding scheme is used internally. Typically a DSP in the electronics inside the HDD takes the raw analog voltages from the read head and uses PRML and Reed–Solomon error correction[144] to decode the sector boundaries…

If this is the case, then why does bit-rot still occur?

Re: Silent Data Corruption Is Real

#127

Post by blablabloe »

mrb wrote: post #12

The exact same silent data corruption issues just happened to my 6 x 5TB ZFS FreeBSD fileserver. But unlike what the poster concluded, mine were caused by bad (ECC!) RAM. I kept meticulous notes, so here is my story... I scrub on a weekly basis. One day ZFS started reporting silent errors on disk ada3, just 4kB: pool: tank state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt wa…

When a double-parity error is detected, the operating system should halt. Maybe that didn't happen properly. Tripple parity errors may go undetected, but how likely is that.

I wonder what 'really' happened.

Re: Silent Data Corruption Is Real

#128

Post by blablabloe »

platosrepublic wrote: post #9

I'm not a database expert, but this seems like something I should worry about, at least a bit. Is this a problem if you store all your persistent data in a database like MySQL?

If you run on proper enterprise gear, the likelyhood of encountering an issue like this is very, very rare.

HP/Dell/SM servers are all ECC memory top to bottom, are properly wired (snark) and SANs have also ECC everywhere. Even on the disk caches.

And in this particular instance, it was basically some private server build being messed up.

Re: Silent Data Corruption Is Real

#129

Post by blablabloe »

3legcat wrote: post #96
tnorgaard wrote: post #63

Earlier quoted context omitted.

Please stop spreading this misinformed statement. I assume you are referring to the ZFS ARC (Adaptive Replacement Cache). It works in much the same way as a regular Linux page cache. It does not take much more memory (if you disable prefetch) and will only use what is available/idle. We use Linux with ZFS on production systems with as low as 1GB memory. We stopped counting the times it has saved the day. :-) ECC is a…

> ECC is nice to have. Actually it seems ECC is important for ZFS filesystems see: http://louwrentius.com/please-use-zfs-with-ecc-memory.html

It's not required, but it doesn't make sense to use ZFS but not to use ECC memory. That's the point. It's like locking the backdoor but leaving the front door wide open.

Re: Silent Data Corruption Is Real

#130

Post by solarengineer »

3legcat wrote: post #96
tnorgaard wrote: post #63

Earlier quoted context omitted.

Please stop spreading this misinformed statement. I assume you are referring to the ZFS ARC (Adaptive Replacement Cache). It works in much the same way as a regular Linux page cache. It does not take much more memory (if you disable prefetch) and will only use what is available/idle. We use Linux with ZFS on production systems with as low as 1GB memory. We stopped counting the times it has saved the day. :-) ECC is a…

> ECC is nice to have. Actually it seems ECC is important for ZFS filesystems see: http://louwrentius.com/please-use-zfs-with-ecc-memory.html

To be clear, it is not ZFS that requires or even mandates ECC. Since ZFS uses data as present in memory and has checks for everything post that, it is prudent to have memory checks at the hardware level.

Thus, if one is using ZFS for data reliability, one ought to use ECC memory as well.

Post reply on HN