Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

nctritech.wordpress.com

Post reply on HN
Print view View original

21–30 of 38 posts

Re: ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

#21

Post by AstralStorm »

random_comment wrote: post #9

This entire article can be summarised as 'guy who has never used ZFS and has no idea whatsoever about how it works writes a critique that exposes their ignorance publicly'. Here's a quote: - “ZFS has CRCs for data integrity A certain category of people are terrified of the techno-bogeyman named “bit rot.” These people think that a movie file not playing back or a picture getting mangled is caused by data on hard driv…

Copy on write is a good thing, as is log structure which is even more resilient. However, it is not strictly superior to journaling in terms of data safety. The copied data will get garbage collected or overwritten after some time and regardless might be tricky to recover.

Re: ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

#22

Post by notacoward »

Totally off base, on several points. Any kind of checksum on the disk only protects what gets to the disk. Filesystem-level CRCs can protect the entire data path. If you have a defect in your RAID card or HBA, or anywhere in the software stack below the filesystem, on-disk CRCs will happily "validate" the already-corrupted data while filesystem-level CRCs are likely to detect the corruption. The author dismisses it as a "remotely likely scenario" but I've seen it happen for real many times. Maybe that's because I have about 3.5x as many years of experience as the author, across what's probably thousands of times as many machines or drives (I've worked on some big system).

The same "I've never seen it so it's not real" fallacy appears again in the discussion of RAID 5. He says that losing a second drive during a rebuild is "statistically very unlikely" but that's not so. Not only have I seen it many times, but the simple math of disk capacities and interface speeds shows that it's not really all that unlikely. I've seen RAID 6 fail because of overlapping rebuild times, leading people to push for more powerful erasure-coding schemes. Over the lifetime of even a medium-sized system, concurrent failures on RAID 5 are likely enough to justify using something stronger.

I was one of the earliest and most outspoken critics of ZFS hype and FUD when it came out. It was and is no panacea, but that doesn't justify more FUD in the other direction to sell backup products or services.

Re: ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

#23

Post by random_comment »

AstralStorm wrote: post #21
random_comment wrote: post #9

This entire article can be summarised as 'guy who has never used ZFS and has no idea whatsoever about how it works writes a critique that exposes their ignorance publicly'. Here's a quote: - “ZFS has CRCs for data integrity A certain category of people are terrified of the techno-bogeyman named “bit rot.” These people think that a movie file not playing back or a picture getting mangled is caused by data on hard driv…

Copy on write is a good thing, as is log structure which is even more resilient. However, it is not strictly superior to journaling in terms of data safety. The copied data will get garbage collected or overwritten after some time and regardless might be tricky to recover.

ZFS has journalling too, in the form of the ZFS Intent Log, where writes are placed rapidly as they occur before becoming part of the main filesystem. The effect is similar to a journal in a journalling filesystem [i.e. to allow recovery and consistency of writes in flight in the event of power loss], but the way it is used is different.

Unless most journaled filesystems, ZFS:

- allows a separate high-performance device to be used for the log. This is important because the cost of journalling can be high when lots of fsyncs are being used to ensure integrity (i.e. try running a write performance test on a database like postgresql using ext4 with and without journalling, you'll see a difference).

- the filesystem log can be mirrored physically, to protect against the risk of log device failure [which would endanger writes in flight].

Other similarities/differences:

In a journalling FS, you need to take the filesystem offline and check the journal. In ZFS, there is continual passive checking of file data and metadata at time of access, as well as the option for an online 'scrub' that is similar to the fsck of a journalled filesystem without requiring dismounting of the filesystem.

While copy-on-write by itself may not be necessarily strictly superior to journalling, ZFS is strictly superior to either.

Re: ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

#24

Post by bbatha »

X86BSD wrote: post #14
Veratyr wrote: post #11

While he's right that it's not as big an issue as ZFS fanatics make it out to be, it _is_ a real issue and they're not just pulling it out their asses. There are a number of studies that actually measured the error rate, some of the bigger ones being done by CERN [0], NetApp [1] and IA (I think there's meant to be a talk or something to go with this one) [2]. ZFS certainly isn't a magic wand you should wave at anythi…

Actually it does replace backups with replication and/or cloning.

Its not backed up until its at least on an external system, ideally in triplicate off-box, off-site, and cold storage. Cloning and replication makes it easier to backup but is no substitute.

Re: ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

#25

Post by AstralStorm »

random_comment wrote: post #23
AstralStorm wrote: post #21

Earlier quoted context omitted.

Copy on write is a good thing, as is log structure which is even more resilient. However, it is not strictly superior to journaling in terms of data safety. The copied data will get garbage collected or overwritten after some time and regardless might be tricky to recover.

ZFS has journalling too, in the form of the ZFS Intent Log, where writes are placed rapidly as they occur before becoming part of the main filesystem. The effect is similar to a journal in a journalling filesystem [i.e. to allow recovery and consistency of writes in flight in the event of power loss], but the way it is used is different. Unless most journaled filesystems, ZFS: - allows a separate high-performance dev…

This is better than RAID patrol reads only in that it also verifies file system structure periodically. And you do not necessarily have to bring down the filesystem to check even when data is in flight as long as it's driver supports online scan functionality. More than one FS does so. (XFS, btrfs and probably JFS. Not ext4 though.)

Not that online scanning makes to much sense anyway. The good filesystems verify sanity of the structure they traverse, so might as well put in a full FS read in cron. Most kinds of damage cannot be repaired on a live filesystem anyway. Even in ZFS.

Re: ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

#26

Post by Veratyr »

X86BSD wrote: post #14
Veratyr wrote: post #11

While he's right that it's not as big an issue as ZFS fanatics make it out to be, it _is_ a real issue and they're not just pulling it out their asses. There are a number of studies that actually measured the error rate, some of the bigger ones being done by CERN [0], NetApp [1] and IA (I think there's meant to be a talk or something to go with this one) [2]. ZFS certainly isn't a magic wand you should wave at anythi…

Actually it does replace backups with replication and/or cloning.

That's not a replacement for backups, that's an implementation of backups and only if you send it to an offline disk or remote system.

Re: ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

#27

Post by X86BSD »

bbatha wrote: post #24
X86BSD wrote: post #14

Earlier quoted context omitted.

Actually it does replace backups with replication and/or cloning.

Its not backed up until its at least on an external system, ideally in triplicate off-box, off-site, and cold storage. Cloning and replication makes it easier to backup but is no substitute.

ZFS send/recv to an offsite ZFS box is a backup. Replacing tape systems. It's incremental. Its compressed. It's faster.

Re: ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

#28

Post by wyoung2 »

AstralStorm wrote: post #18
rgbrenner wrote: post #10

Earlier quoted context omitted.

Just asserting "CRCs are useless" is putting a lot of trust on stuff that has real-world failure modes. Yes, and he does this over and over again throughout the article. I have personally experienced at least 3 scenarios that he has determined won't happen. If this guy wrote a filesystem (something that he pretends to have enough experience to critique), it would be an unreliable unusable piece of crap.

You have worse problems that a filesystem won't catch if RAM gets randomly corrupted. Including said CRC check itself getting corrupted or code writing putt data structures to disk being wrong. Neither of those is caught by CRC better than by a dirty bit. It so happens that journaling file systems already have a degree of redundancy for writes built into them unless you defeat it.

The trivial case is that the data is corrupted in RAM prior to being written. If we take the simple case of a 2-disk mirror, the same wrong data is going to be written to both disks, the checksums will match, and the filesystems and underlying disks will be oblivious to the problem. ZFS can't help here, but neither can RAID-5.

The far more risky situations involve reading back data.

A properly-optimized RAID or RAID-like system will read half the blocks from one disk and half from the other when dealing with a 2-disk mirror.

With RAID-1, if the data blocks read cleanly from one disk — that is, the hard disk's ECC does its thing, as the author expects — but the data bytes are then corrupted in RAM during the DMA transfer, RAID won't detect the problem. Your application will simply have errors in those blocks, and it'll be oblivious to the problem unless there is some corruption detection ability in the data format.

With a ZFS mirror, things are different. If the blocks are cleanly read from the disk (again according to those in-drive ECC checks) but the bytes are corrupted during the DMA transfer to RAM, ZFS will detect it, because it always double-checks the hashes — cryptographycally-strong hashes, mind, not CRCs, as the author misstates — after reading the data in from disk. This will cause ZFS to attempt a second read from the corresponding block in the other side of the mirror. Assuming you don't get a second RAM corruption, the checksum will match this time, so ZFS will re-write the clean block to the first disk. ZFS is incorrectly assuming it was the drive that corrupted the block, but it doesn't matter because all that happens is a correct block is overwritten with the same correct block.

Now let's take a trickier case. What if your RAM is so flaky that it re-corrupts the clean block on its way back out to the first disk during this unnecessary re-write? ZFS will write the correct checksum along with that block's data, so that when it comes time to re-read that block, the checksum won't match the data. It doesn't matter whether the RAM corrupts the checksummed data or the checksum itself, because the odds are astronomically against both being corrupted in a way that causes the two to match. When ZFS is told to re-read that corrupted block, either by the application or by a background scrub, it will again decide it needs to overwrite the first disk's copy of the block with the copy from the second disk, which this time is in fact corrupted on-disk. Unless your RAM corrupts the data a third time, this time it will write the correct data to disk.

RAID can't do any of that. All RAID can do is say, "These two blocks don't match each other, but both have good on-disk ECC, so PANIC." Different RAID implementations do different things here. Some will just mark the array as degraded and force the operator to choose one disk to mirror back onto the other. If the operator guesses wrong, you've got two copies of the bad data now.

ZFS doesn't have to guess: it knows which copy is wrong with astronomical odds in favor of being correct.

Re: ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

#29

Post by wyoung2 »

asveikau wrote: post #5

A few years ago I had a drive at home that was flipping bits, randomly corrupting my files. It inspired me to build a ZFS disk server and introduce redundancy in my home setup. A bunch of this article reads as if this scenario, which I in fact hit, won't happen, drives do it better, etc. But it happens. It happened to me. The drive did not "magically fix itself", and instead got worse over time. With ZFS, if it happe…

I've got a ZFS server here that regularly detects some small number of megs of incorrect data on each week's scrub. This week, it happens to be 4.28M. Every week, ZFS finds the correct copy and fixes it.

I have no idea what the problem is with this server. There are no SMART failures or kernel messages indicating hardware failure, and the system doesn't hard-crash. The thing is, I don't actually have to care, because ZFS is actively taking care of the problem. Until one of the disks goes so bad that SMART or the kernel's SATA layer or ZFS can point me at it, I can just passively let ZFS continue protecting me.

If this were a RAID, the first risk is that the RAID system wouldn't have a scrub command at all. Some do, but not all. Without such a command, those on-disk ECCs the author heaps so much praise on won't help him. I've got the same ECCs backing my ZFS, and clearly the data is getting corrupted anyway, somehow.

Let's keep the author's context in mind, which is apparently that we're going to use motherboard or software RAID, since he's budgeted $0 for a hardware RAID card, so the chances are higher that there is no scrub or verify command.

If our RAID implementation does happen to have a scrub or verify command, it might be forced to just kick one of the disks out or mark the whole array as degraded, depending on where in the chain the corruption happened. If it does that, it'll take a whole lot longer to rewrite one of the author's cheap 3 TB disks than it took ZFS on my file server to fix the few megs of corrupted blocks.

And that's not all. I have a second anecdote, the plural of which is "data," right? :)

Another ZFS-based system I manage had a disk die outright in it. SMART errors, I/O timeouts, the whole bit. Very easy to diagnose. So, I attached a third disk in an external hard disk enclosure to the pained ZFS mirror, which caused ZFS to start resilvering it.

Before I go on, I want to point out that this shows another case where ZFS has a clear advantage. In a typical hardware RAID setup, a 2-disk mirror is more likely to be done with a 2-port RAID card, because they're cheaper than 4-port and 8-port cards. That means there is a very real chance that you couldn't set up a 3-disk mirror at all, which means you're temporarily reduced to no redundancy during the resilver operation. Even if you've got a spare RAID port on the RAID card or motherboard, you might not have another internal disk slot to put the disk in. With ZFS, I don't need either: ZFS doesn't care if two of a pool's disks are in a high-end RAID enclosure configured for JBOD and the third is in a cheap USB enclosure.

The point of having a temporary 3-disk mirror is that the dying disk wasn't quite dead yet. That means it was still useful for maintaining redundancy during the resilvering operation. With the RAID setup, you might be forced to replace the dying disk with the new disk, which means you lose all your redundancy during the resilver.

Now as it happens, sometime during the resilver operation, `zfs status` began showing corruptions. ZFS was actively fixing them like a trooper, but this was still very bad. It turned out that the cheap USB external disk enclosure I was using for the third disk was flaky, so that when resilvering the new disk, it wasn't always able to write reliably. I unmounted the ZFS pool, moved the new disk to a different external USB disk enclosure, re-mounted the pool, and watched it pick the resilvering process right back up. Once that was done, I detached the dying disk from the mirror and did a scrub pass to clear the data errors, and I was back in business having lost no data, despite the hardware actively trying to kill my data twice over.

There are still cases where I'll use RAID over ZFS, but I'm under no illusions that ZFS has no real advantages over RAID. I've seen plenty of evidence to the contrary.

Re: ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot

#30

Post by wyoung2 »

AstralStorm wrote: post #19
Mindless2112 wrote: post #7

As someone who has lost some files to a silently malfunctioning hard disk in the past, I think I'll stick with ZFS. Checksumming, RAID-Z, and periodic scrubbing would have saved my files. Even having backups did not -- after all, what good is a bit-for-bit copy of a corrupted file? (On a side note, ZFS -- at least OpenZFS -- doesn't support any CRC algorithms for use as its checksum.)

Mostly periodic scrubbing and patrol reads I reckon. Which is as required with RAID without ZFS.

Scrub/verify/patrols, whatever you want to call it, with RAID all it can do is say, "Well shit, these two copies don't match. What do you want me to do about it, boss?"

ZFS doesn't have to guess which copy is wrong. It knows, and it will automatically replace it.

More, ZFS will even do this on a ZFS mirror when reading half the data blocks from one disk and half from the other, because it reads the cryptographically-strong checksums in with each data block and checks them before delivering the data to the application. If the checksum doesn't match, it rewrites that block from the redundant copy on the other disk(s).

RAID can't do that. If one of a mirror's data blocks is corrupted on disk but with a correct ECC, so that the two blocks don't match but both read cleanly, RAID can't tell which one is correct, so it'll typically just force the system administrator to choose one disk to overwrite the other with. That exchanges astronomical odds against incorrect data for coin flip odds against.

Post reply on HN