Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: ZFS silent data corruption bugfix – my research results

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 90 posts

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#21

Post by nightfly »

antongribok wrote: post #15
buildbot wrote: post #10

Earlier quoted context omitted.

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

The reason, and the difference, is that all these other filesystems have check and repair (and sometimes multiple) tools. Please correct me, but ZFS has none.

Zfs has checksumming and scrubs, which can catch lots of data corruption that (most?) other filesystems can't catch catch at all

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#22

Post by antongribok »

sandreas wrote: post #11
antongribok wrote: post #4

Earlier quoted context omitted.

For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me. I finally moved to btrfs earlier this year, and so far I'm glad I did. I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Ta…

Does btrfs support native encryption?

Btrfs does not have native encryption.

I use LUKS / dm-crypt on entire drive partitions. Did that with ZFS too, and by the time ZFS got this feature I was already planning my migration to btrfs.

I like encrypting with LUKS, because I can have a drive configured with mine and my wife's password. Either one of us can take that drive, plug it in, and Gnome will put a nice graphical prompt asking for a password, then decrypt the drive and mount the filesystem.

If I get hit by a bus, LUKS makes it much easier for my family to get access to important data without having to have that data sit around somewhere in plaintext.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#23

Post by antongribok »

michaelmrose wrote: post #18
antongribok wrote: post #4

Earlier quoted context omitted.

For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me. I finally moved to btrfs earlier this year, and so far I'm glad I did. I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Ta…

Isn't raid5 on raid5 perpetually broken and unsafe since inception? Also a complicating factor with kernel upgrades is that while zfs release notes clearly delineate what kernel versions are supported that information doesn't appear to be meaningfully encoded in package metadata so if you use new enough kernels compared to the version of zfs for your distro it is possible to front run support. For instance 2.2.2 supp…

The packaging was a major reason for switching to btrfs. I run sudo dnf upgrade and that's it, my system is upgraded. Zero issues ever. With ZFS I had to pin to older kernel versions, not to mention a bunch of manual steps and cleanup after any major version upgrade (every 6-12 months).

Re btrfs raid 5/6, yes everyone knows about this, and this is why I have it only on my backup system. My primary data which holds 15TB of family photos and videos is on raid 1. The offsite is there only for the time my house burns down.[1]

[1] I watched my neighbor's house go up in flames 2 years ago, and it finally got me going on setting up remote backups. The fire spread to 3 other houses, and everything happened very, very quickly. No one got hurt, but multiple families got displaced for more than a year. Besides having backups , it's also a good reminder to have adequate insurance. One neighbor did not.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#24

Post by chungy »

antongribok wrote: post #15
buildbot wrote: post #10

Earlier quoted context omitted.

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

The reason, and the difference, is that all these other filesystems have check and repair (and sometimes multiple) tools. Please correct me, but ZFS has none.

What good would ZFS be without that?

The key differences are in two places:

1. Every administration task for ZFS is done online. You don't need to take your pool offline just to check for errors and repair them (if possible). (Mind that on FreeBSD, UFS can usually have a fsck done in the background while the file system is in-use. Just about zero Linux-native file systems have this capability)

2. "other filesytems" can only hope to detect and repair inconsistencies in their metadata structures. If your files are corrupted, they can't know and won't tell you that they are. ZFS checksums everything, including regular file data. It will repair regular file data too, if possible.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#25

Post by antongribok »

nightfly wrote: post #21
antongribok wrote: post #15

Earlier quoted context omitted.

The reason, and the difference, is that all these other filesystems have check and repair (and sometimes multiple) tools. Please correct me, but ZFS has none.

Zfs has checksumming and scrubs, which can catch lots of data corruption that (most?) other filesystems can't catch catch at all

I guess I meant it as having a separate repair tool, but yes, you're of course correct.

To me it was always weird to not have a separate tool for being able to do an offline ZFS repair.

With regards to data corruption, I mean, this is exactly why I moved to btrfs, because it's able to catch bit rot same as ZFS.

I think ultimately this is why ZFS data corruption bugs always are such a big deal... It's because in a lot of tech circles ZFS is put on this infallible pedestal where it can never do any harm to your data.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#26

Post by denkmoon »

godelski wrote: post #8

Is there any good beginner friendly documentation for zfs? I've started using it as a testing/learning NAS with a raspberry pi (cloning my google library to immich). It has not been a clearly easy process and errors are very much not clear. I recently extended a single drive to 2 and now I can't import due to corrupted metadata and reports bad disk but smartctl reports all fine. Stack overflow is all over the place a…

Truenas forum. Level 1 Techs youtube channel. Lawrence Systems youtube channel. the man pages of course.

First thing I'd suggest is really getting the terminology under your belt. Everything makes so much more sense when you use the correct terminology. For example, you don't extend a drive. That doesn't really make sense. I suspect what you mean is that you added a vdev to a zpool. Don't think about things in terms of disks and extending them, think about things in terms of one or more disks making up a vdev, and a zpool is, well, a pool of those vdevs. Zfs then works out how best to write data across the vdevs in the pool. Not disks.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#27

Post by ikiris »

sandreas wrote: post #11
antongribok wrote: post #4

Earlier quoted context omitted.

For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me. I finally moved to btrfs earlier this year, and so far I'm glad I did. I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Ta…

Does btrfs support native encryption?

its also had many data loss / corruption issues, and still looks to be scary for years to come.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#28

Post by godelski »

denkmoon wrote: post #26
godelski wrote: post #8

Is there any good beginner friendly documentation for zfs? I've started using it as a testing/learning NAS with a raspberry pi (cloning my google library to immich). It has not been a clearly easy process and errors are very much not clear. I recently extended a single drive to 2 and now I can't import due to corrupted metadata and reports bad disk but smartctl reports all fine. Stack overflow is all over the place a…

Truenas forum. Level 1 Techs youtube channel. Lawrence Systems youtube channel. the man pages of course. First thing I'd suggest is really getting the terminology under your belt. Everything makes so much more sense when you use the correct terminology. For example, you don't extend a drive. That doesn't really make sense. I suspect what you mean is that you added a vdev to a zpool. Don't think about things in terms…

Thanks. I'll look into these. Seems like a rough entry, but I can totally get it being "obvious" after you get through. I'm a vim user so I can do this hahaha

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#29

Post by michaelmrose »

antongribok wrote: post #23
michaelmrose wrote: post #18

Earlier quoted context omitted.

Isn't raid5 on raid5 perpetually broken and unsafe since inception? Also a complicating factor with kernel upgrades is that while zfs release notes clearly delineate what kernel versions are supported that information doesn't appear to be meaningfully encoded in package metadata so if you use new enough kernels compared to the version of zfs for your distro it is possible to front run support. For instance 2.2.2 supp…

The packaging was a major reason for switching to btrfs. I run sudo dnf upgrade and that's it, my system is upgraded. Zero issues ever. With ZFS I had to pin to older kernel versions, not to mention a bunch of manual steps and cleanup after any major version upgrade (every 6-12 months). Re btrfs raid 5/6, yes everyone knows about this, and this is why I have it only on my backup system. My primary data which holds 15…

I wonder if a metapackage that always depends on kernel <= supported would resolve the issue by ensuring you don't need to pin a specific version manually.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#30

Post by AndrewDavis »

antongribok wrote: post #25
nightfly wrote: post #21

Earlier quoted context omitted.

Zfs has checksumming and scrubs, which can catch lots of data corruption that (most?) other filesystems can't catch catch at all

I guess I meant it as having a separate repair tool, but yes, you're of course correct. To me it was always weird to not have a separate tool for being able to do an offline ZFS repair. With regards to data corruption, I mean, this is exactly why I moved to btrfs, because it's able to catch bit rot same as ZFS. I think ultimately this is why ZFS data corruption bugs always are such a big deal... It's because in a lot…

Could you please explain what you mean by a separate repair tool kit?
Post reply on HN