Live data from Hacker News

Silent Data Corruption Is Real

changelog.complete.org

41–50 of 154 posts

Re: Silent Data Corruption Is Real

#41
post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

Having used ZFS on Solaris/x86 for years and years, and now on Linux, I would say that your best best is to use ZFS.

While ZFS might not be supported oput of the box on "your-choice-distro" I would point out that Ansible or other automation tools should make it pretty easy to end up with a repeatable installed machine or VM image with ZFS.

Re: Silent Data Corruption Is Real

#42
post #24
post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

If you actually care about your data you simply will have to switch platforms. Illumos or FreeBSD. Linux has no answer to the horrors of garbage Filesystems and garbage lying hardware, firmware etc that has plagued UNIX admins for ever. Btrfs is apparently the best they can do and its glaringly deficient architecturally. Filesystems are HARD. And as bcantrill notes, if you can't get simple things like epoll right you…

The question is whether or not "the market" cares enough about "its data" enough that they are willing to pay for what would cost. And data checksumming and RAID 1 (so you can recover from the data corruption) does cost something --- even if it is a 100% expansion in the bytes needed to store the data. And COW file systems do cost something in terms of HDD overhead. Maybe you have enough slack in terms of how you are using your HDD that you don't notice, but if you are using all of your HDD's I/O capacity with an update-in-place file system, when you switch to a COW file system, you will notice the performance hit.

If your only copy of your children's baby pictures are on a solo 4TB drive in your house --- then you may be more than willing to pay that cost. But what if your house burns down? It may be that the right answer is do data backups in the cloud, and there you will want erasure coding, and perhaps more than 100% overhead --- you might erasure coding scheme that has a 150% to 300% blowup in space so you have better data availability, not just data loss avoidance.

I do agree that file systems are hard, but at the same time you need to have a business case for doing that level of investment. This is true whether it is for a proprietary or open source code base. Many years ago, before ZFS was announced, I participated in a company wide study at my employer at the time about whether it was worth it to invest in file systems. Many distinguished engineers and fellows, as well as business people and product managers participated. The question was studied not just in terms of the technical questions, but also from the business perspective --- "was the ROI on this positive; would customers actually pay more such that the company would actually gain market share, or otherwise profit from making this investment". And the answer was "No". This made me sad, because it meant that my company wasn't going to invest in further file system technologies. But from a stock holder's perspective, the company's capital was better spent investing in middleware and database products, because there the ROI was actually positive.

From everything that I've read and from listening to Bryan's presentations, I understanding is that at Sun they did _not_ do a careful business investigation before deciding to invest in ZFS. And so, "Good for Open Solaris!" Maybe it kinda sucked if you were a Sun shareholder, but maybe Sun was going to go belly-up anyway, so might as well get some cool technology developed before the ship went under. :-)

As far as Linux is concerned, at some level, the amount of volunteer manpower and corporate investment in btrfs speaks to, I suspect, a similar business decision being made across the Linux ecosystem about whether or not the ROI of investing in btrfs makes sense. The companies that have invested in ext4 because in no-journal mode, have done so because if what you want is a back-end, local disk file system, on top of which you put a cluster file system like HDFS or GFS or Colossus, and where the data integrity is done end-to-end at the cluster file system level, and not at the local disk file system, you'll want the lowest overhead file system layer you can get. That doesn't mean that you don't care about data integrity; you do! But you've made an architectural decision about where to place that functionality, and a business decision about where to invest their proprietary and open source development work.

Each company and each user should make their own decisions. If you don't need the software compatibility and other advantages of Linux, and if data integrity is hugely important, and you don't want to go down the path of using some userspace solution like Camilstore, or some cloud service like AWS S3 or Google Compute Storage, then perhaps switching to FreeBSD is the right solution for you. Or you can choose to contribute to make btrfs more stable. And in some cases that may mean being willing to accept a certain risk for data loss at one level, because you handle your integrity requirements at a different level. (e.g., why worry about your source tree on your laptop when it's backed up regularly via "git push" to servers around the internet?)

Re: Silent Data Corruption Is Real

#43
ZFS is also crazy good on surviving disks with bad sectors (as long as they still respond fast). Check out this paper: https://research.cs.wisc.edu/wind/Publications/zfs-corruptio...

They even spread the metadata across the disk by default. I'm running on some old WD-Greens with 1500+ of bad sectors and it's cruising along with RAIDZ just fine.

There is also failmode=continue where ZFS doesn't hang when it can't read something. If you have a distributed layer above ZFS that also checksums (like HDFS) you can go pretty far even without RAID and quite broken disks. There is also copies=n. When ZFS broke, the disk usally stopped talking or died a few days later. btrs, ext4 just choke and remount ro quite fast (probably the best and correct course of action) but you can tell ZFS to just carry on! Great piece of engineering!

Re: Silent Data Corruption Is Real

#44
post #3

It really bugs me (and has for a while) that there is still no mainstream linux filesystem that supports data block checksumming. Silent corruption is not exactly new, and the odds of running into it have grown significantly as drives have gotten bigger. It's a bit maddening that nobody seems to care (or maybe I'm just looking in the wrong places) (...sure, you could call zfs or btrfs "mainstream", I suppose, but whe…

Making data on individual drives is probably not a long term goal for integrity. It is more likely that multi homing data in geographically disjoint locations with some sort of syncing is a better long term goal. Fixing silent data corruption on a single drive doesn't solve any of the much more likely disasters, like fire, flooding, weather, etc. Not even datacenters can withstand lightning.

Not even datacenters can withstand lightning.

Data centers do have lighting protection. Lighting protection is very low-tech.

https://en.m.wikipedia.org/wiki/Lightning_rod

Re: Silent Data Corruption Is Real

#45
I know for sure that btrfs scrub found 8 correctable errors on my home server filesystem last July. This is obviously great news for me. Contrary to a lot of people here I've personally found btrfs to be really stable (as long as you don't use raid5/6 though).

Re: Silent Data Corruption Is Real

#46

Earlier quoted context omitted.

Making data on individual drives is probably not a long term goal for integrity. It is more likely that multi homing data in geographically disjoint locations with some sort of syncing is a better long term goal. Fixing silent data corruption on a single drive doesn't solve any of the much more likely disasters, like fire, flooding, weather, etc. Not even datacenters can withstand lightning.

The problem with multiple geographic locations, in this context, is that you'd have to read from all of them, and compare the results, to know that you have file corruption. Which is, needless to say, not something that it makes sense to do. The purpose of data block checksumming isn't to make your data more resilient (at least directly), it's to make sure you know you have a problem. Once you know you have a problem…

Agreed, and that's how zfs does it behind the scenes. When it detects a read error, it uses the pool redundancy (whether mirrors or RAIDZn, whichever you're using) to transparently recover from the error. Even if you set up a pool with only one disk you can still set it to keep multiple copies of all the data.

A super-ZFS that automatically did that using remote mirrors would be interesting, but would also stretch the definition of "transparent" a bit.

Re: Silent Data Corruption Is Real

#47
post #37
post #28

Earlier quoted context omitted.

ZFS not portable? Have you ever used it? "ZFS export" on the host system, remove drives, insert drives in new server, "ZFS import" on the new server, It really is that simple.

I think he means portability of the software. What I want to 'ZFS import' on a Windows or macOS machine?

Oh, well it's ported from Solaris to illumos and FreeBSD and to a lesser extent OS X and Linux. So I'm still confused about the portability claim.

Re: Silent Data Corruption Is Real

#48
post #37

Earlier quoted context omitted.

I think he means portability of the software. What I want to 'ZFS import' on a Windows or macOS machine?

ZFS on OSX has been revived I believe.

I wonder if the recent Linux syscall emulation on Windows would somehow make it possible or easier to port ZFS on Linux to Windows.

I know you have the SPL anyway, so maybe with the addition of the Linux POSIX-ish layer in there this could be the case...

Re: Silent Data Corruption Is Real

#49
post #6

Earlier quoted context omitted.

Btrfs has experienced some data loss bugs in recent memory. It looks like ZFS is the only remaining option. https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Da... https://www.spinics.net/lists/linux-btrfs/msg59190.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...

Any idea if ZFS plays well with Ubuntu's full-disk encryption? I've used the FDE option at install for years and every time I upgrade (I wipe & reinstall every year or so) I try to understand how to first set up ZFS, then FDE, and then I realize it's far too complicated for me. Any good tutorials or setup guides that even a moron could understand? I've got a pretty good setup now with a fairly complex fstab, multiple…

You can always use zvols and just use ext4+luks on that. There is also work on ZFS native encryption that looks pretty promising, not sure if it's ready yet.

Re: Silent Data Corruption Is Real

#50
post #6

Earlier quoted context omitted.

Btrfs has experienced some data loss bugs in recent memory. It looks like ZFS is the only remaining option. https://www.phoronix.com/scan.php?page=news_item&px=Btrfs-Da... https://www.spinics.net/lists/linux-btrfs/msg59190.html https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg...

Any idea if ZFS plays well with Ubuntu's full-disk encryption? I've used the FDE option at install for years and every time I upgrade (I wipe & reinstall every year or so) I try to understand how to first set up ZFS, then FDE, and then I realize it's far too complicated for me. Any good tutorials or setup guides that even a moron could understand? I've got a pretty good setup now with a fairly complex fstab, multiple…

I haven't tried to do it with Ubuntu specifically, but I do know that ZFS-on-linux works fine atop LUKS full-disk encryption. My laptop is running NixOS with such a setup, and I'm pretty sure I followed the Ubuntu ZFS documentation while figuring out how to do it.

(summary: I don't know how to make it super easy, but what you want should totally work)

Post reply on HN