Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: ZFS silent data corruption bugfix – my research results

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 90 posts

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#11

Post by sandreas »

antongribok wrote: post #4
KyleSanderson wrote: post #2

https://github.com/openzfs/zfs/pull/15529#pullrequestreview-... Honestly, ZFS is the best thing on the (Free)BSDs only... On Linux it doesn't even use the page cache, and you conflict severely with L2ARC. I know there's a variety of people who don't care, but still for real users it's not an actual option.

For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me. I finally moved to btrfs earlier this year, and so far I'm glad I did. I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Ta…

Does btrfs support native encryption?

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#12

Post by sandreas »

AndrewDavis wrote: post #7

This bug shouldn't really scare people. It's requires such an incredibly specific workload to hit Here's a post by RobN (the dev who wrote the fix) on the ZFS On Linux mailing list > There's a really important subtlety that a lot of people are missing in this. The bug is _not_ in reads. If you read data, its there. The bug is that sometimes, asking the filesystem "is there data here?" it says "no" when it should say…

This is a great explanation, thank you.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#13

Post by 1letterunixname »

I come from the GPFS, Lustre, and Panasas world of HPCC.

Personally, I used (past tense) ZoL in 2014-2017 on Ubuntu. The issue is that the array eventually entered an unrecoverable state where it could no longer be mounted RW. That wasn't the end of the world, but the support from ZoL was to shrug at it. That was the end of that because without support and without pride, something that appears shiny is effectively useless.

Been running many XFS volumes over mdadm RAID10 near-2 arrays. Zero major problems in 5+ years with over 400 TiB online. SGI's, Redhat's, and more contributions to various Linux storage components are excellent.

My conclusion is that ZoL != Solaris ZFS. Once it left enterprise with controlled hardware and dedicated engineering & support teams, it regressed and devolved. Beware of fanboys where passion and tribalism exceeds evidence and reliability.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#14

Post by tarruda »

antongribok wrote: post #4
KyleSanderson wrote: post #2

https://github.com/openzfs/zfs/pull/15529#pullrequestreview-... Honestly, ZFS is the best thing on the (Free)BSDs only... On Linux it doesn't even use the page cache, and you conflict severely with L2ARC. I know there's a variety of people who don't care, but still for real users it's not an actual option.

For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me. I finally moved to btrfs earlier this year, and so far I'm glad I did. I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Ta…

> Replication is with rsync and borg, not snapshots.

I backup my files with borg, but I still use snapshots during the Borg backup to ensure files are not modified during the process

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#15

Post by antongribok »

buildbot wrote: post #10
AndrewDavis wrote: post #7

This bug shouldn't really scare people. It's requires such an incredibly specific workload to hit Here's a post by RobN (the dev who wrote the fix) on the ZFS On Linux mailing list > There's a really important subtlety that a lot of people are missing in this. The bug is _not_ in reads. If you read data, its there. The bug is that sometimes, asking the filesystem "is there data here?" it says "no" when it should say…

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

The reason, and the difference, is that all these other filesystems have check and repair (and sometimes multiple) tools.

Please correct me, but ZFS has none.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#16

Post by antongribok »

tarruda wrote: post #14
antongribok wrote: post #4

Earlier quoted context omitted.

For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me. I finally moved to btrfs earlier this year, and so far I'm glad I did. I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Ta…

> Replication is with rsync and borg, not snapshots. I backup my files with borg, but I still use snapshots during the Borg backup to ensure files are not modified during the process

Yeah, of course you should be doing that. I was just trying to say that people shouldn't pretend like having snapshot-based, filesystem-level replication alone (which can be a very efficient way of replicating data changes) is a good backup strategy.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#18

Post by michaelmrose »

antongribok wrote: post #4
KyleSanderson wrote: post #2

https://github.com/openzfs/zfs/pull/15529#pullrequestreview-... Honestly, ZFS is the best thing on the (Free)BSDs only... On Linux it doesn't even use the page cache, and you conflict severely with L2ARC. I know there's a variety of people who don't care, but still for real users it's not an actual option.

For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me. I finally moved to btrfs earlier this year, and so far I'm glad I did. I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Ta…

Isn't raid5 on raid5 perpetually broken and unsafe since inception?

Also a complicating factor with kernel upgrades is that while zfs release notes clearly delineate what kernel versions are supported that information doesn't appear to be meaningfully encoded in package metadata so if you use new enough kernels compared to the version of zfs for your distro it is possible to front run support. For instance 2.2.2 supports up to 6.6 but you could very well for instance install 6.7 and it might not work.

The somewhat broken thing is not encoding known data like which kernel is supported to automatically do the right thing in the packaging system not the filesystem. The lazy fix is to just manually handle kernel updates. The lazier one is to grab the release notes and update if latest kernel is <= supported.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#19

Post by gavinhoward »

antongribok wrote: post #15
buildbot wrote: post #10

Earlier quoted context omitted.

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

The reason, and the difference, is that all these other filesystems have check and repair (and sometimes multiple) tools. Please correct me, but ZFS has none.

You're completely wrong.

ZFS's scrub is both a check and a repair tool. It's already saved some of my data.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#20

Post by [deleted] »

antongribok wrote: post #15
buildbot wrote: post #10

Earlier quoted context omitted.

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

The reason, and the difference, is that all these other filesystems have check and repair (and sometimes multiple) tools. Please correct me, but ZFS has none.

[deleted]
Post reply on HN