Live data from Hacker News

Tell HN: ZFS silent data corruption bugfix – my research results

news.ycombinator.com

31–40 of 90 posts

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#31

Earlier quoted context omitted.

The reason, and the difference, is that all these other filesystems have check and repair (and sometimes multiple) tools. Please correct me, but ZFS has none.

You're completely wrong. ZFS's scrub is both a check and a repair tool. It's already saved some of my data.

You are partly right. Zfs scrub is a repair tool when it has parity / mirrored copy of data to recreate it.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#32
post #27

Earlier quoted context omitted.

Does btrfs support native encryption?

its also had many data loss / corruption issues, and still looks to be scary for years to come.

I've never had data loss with either ZFS or btrfs.

For me on Fedora, uptime is better and maintenance overhead is lower on btrfs.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#34

This bug shouldn't really scare people. It's requires such an incredibly specific workload to hit Here's a post by RobN (the dev who wrote the fix) on the ZFS On Linux mailing list > There's a really important subtlety that a lot of people are missing in this. The bug is _not_ in reads. If you read data, its there. The bug is that sometimes, asking the filesystem "is there data here?" it says "no" when it should say…

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

My experience has been aggravated by being simultaneously affected by https://github.com/openzfs/zfs/issues/11893, which results in all zfs operations hanging (kill -9 is ineffective) until a power cycle. This round of upgrades was not fun.

Debian bookworm users: You need to enable backports for zfs. Both bugs are fixed in backports but still present in stable.

Let's do away with the tribalism in both ways? Calling it perfect is quite a stretch. I'm not saying "don't use zfs", just "you want to be prepared for the worst-case if your data matters". 3-2-1 and "validate your backups" still apply.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#35

This bug shouldn't really scare people. It's requires such an incredibly specific workload to hit Here's a post by RobN (the dev who wrote the fix) on the ZFS On Linux mailing list > There's a really important subtlety that a lot of people are missing in this. The bug is _not_ in reads. If you read data, its there. The bug is that sometimes, asking the filesystem "is there data here?" it says "no" when it should say…

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

No, you're just hearing about ZFS corruption like you've heard corruption in other filesystems, this wont be the first and last.

Its easy to feel like its being targeted with some kind of campaign, however the truth is rarely that exciting. The software is being used by more people, which means it will expose more bugs.

No software is perfect, it just now starting to be abused enough to be important enough to be talked about. This stage is entirely normal.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#36

Earlier quoted context omitted.

You're completely wrong. ZFS's scrub is both a check and a repair tool. It's already saved some of my data.

You are partly right. Zfs scrub is a repair tool when it has parity / mirrored copy of data to recreate it.

It's safe to make the assumption that the tool isn't magic.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#37
post #34

Earlier quoted context omitted.

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

My experience has been aggravated by being simultaneously affected by https://github.com/openzfs/zfs/issues/11893 , which results in all zfs operations hanging (kill -9 is ineffective) until a power cycle. This round of upgrades was not fun. Debian bookworm users: You need to enable backports for zfs. Both bugs are fixed in backports but still present in stable. Let's do away with the tribalism in both ways? Calling…

Perfect for me is not perfect for others of course. I have not run into the issue you linked, that sounds frustrating to deal with.

Backups is the only (near) 100% way to insure you data I agree.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#38

Earlier quoted context omitted.

You're completely wrong. ZFS's scrub is both a check and a repair tool. It's already saved some of my data.

You are partly right. Zfs scrub is a repair tool when it has parity / mirrored copy of data to recreate it.

A scrub can also repair data when using `ncopies` greater than one even outside any mirrors or parity.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#39

Earlier quoted context omitted.

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

No, you're just hearing about ZFS corruption like you've heard corruption in other filesystems, this wont be the first and last. Its easy to feel like its being targeted with some kind of campaign, however the truth is rarely that exciting. The software is being used by more people, which means it will expose more bugs. No software is perfect, it just now starting to be abused enough to be important enough to be talk…

But it's totally different than corruption in other filesystems. People are acting like upgrading to ZFS 2.2.0 ate all their data like XFS used to back in the bad old days. I remember once the power went out at my house, and the entire XFS filesystem was irreparably damaged.

This bug is super hard to trigger, and has roots back to 2006 and only surfaced with a recent coreutils update. Yet there are numerous posts in the past few days verging on hysteria. I myself was very worried until I saw posts like the grand parent comment or the linked GitHub issues.

I don't believe there's actually a campaign of ZFS haters sitting around in a storeroom somewhere posting to phoronix and HN, but it does feel like a lot of strongly held opinions are being unleashed now that ZFS had this bug.

I'm pretty sure ZFS has been abused a lot - ten years ago I was helping test petabyte scale ZFS clusters, doing tons of concurrent IO and such. I'm pretty sure Sun and Oracle tested the crap out of it.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#40
Old geezer take on the situation:

Back in (say) the 1980's, the code for a serious OS's filesystem was small enough that formal verification was at least imaginable. Similar for writing test code for ~all of the corner cases.

Not now.

When and where you value reliability - try to keep things simple, stick to code paths well-tested by the passage of the masses, do at least some testing of your own, and have a Plan B.

Post reply on HN