Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: ZFS silent data corruption bugfix – my research results

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 90 posts

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#41

Post by qwertox »

AndrewDavis wrote: post #7

This bug shouldn't really scare people. It's requires such an incredibly specific workload to hit Here's a post by RobN (the dev who wrote the fix) on the ZFS On Linux mailing list > There's a really important subtlety that a lot of people are missing in this. The bug is _not_ in reads. If you read data, its there. The bug is that sometimes, asking the filesystem "is there data here?" it says "no" when it should say…

>> So while I would never say that no one has ever hit the problem unknowingly, I feel pretty confident that they haven't. And if you're not sure, ask yourself if you've ever had highly parallel workloads that involve writing and seeking the same files at the same moment.

It makes it sound unlikely, but if I have a couple of VMs in datasets (all formatted as ext4 internally and some running DBs inside them) each is one big `raw` file which is getting a lot of reads and writes, I assume.

How are they at risk?

Also, what about ZVOLs mounted as ext4 drives in these VMs?

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#42

Post by moss2 »

buildbot wrote: post #10
AndrewDavis wrote: post #7

This bug shouldn't really scare people. It's requires such an incredibly specific workload to hit Here's a post by RobN (the dev who wrote the fix) on the ZFS On Linux mailing list > There's a really important subtlety that a lot of people are missing in this. The bug is _not_ in reads. If you read data, its there. The bug is that sometimes, asking the filesystem "is there data here?" it says "no" when it should say…

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

I'm unreasonably annoyed you used etcetera instead of et cetera.

'Et' is Latin for 'and'. The term means "and other things", not "andotherthings".

No, I have nothing important to add to the ZFS discussion.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#43

Post by worthless-trash »

buildbot wrote: post #39
worthless-trash wrote: post #35

Earlier quoted context omitted.

No, you're just hearing about ZFS corruption like you've heard corruption in other filesystems, this wont be the first and last. Its easy to feel like its being targeted with some kind of campaign, however the truth is rarely that exciting. The software is being used by more people, which means it will expose more bugs. No software is perfect, it just now starting to be abused enough to be important enough to be talk…

But it's totally different than corruption in other filesystems. People are acting like upgrading to ZFS 2.2.0 ate all their data like XFS used to back in the bad old days. I remember once the power went out at my house, and the entire XFS filesystem was irreparably damaged. This bug is super hard to trigger, and has roots back to 2006 and only surfaced with a recent coreutils update. Yet there are numerous posts in…

> But it's totally different than corruption in other

> filesystems. People are acting like upgrading to ZFS 2.2.0

> ate all their data like XFS used to back in the bad old

> days. I remember once the power went out at my house, and

> the entire XFS filesystem was irreparably damaged.

Different bugs manifest differently due to the coding, this is just the same.

ZFS wouldn't even have a hundredth of the testing of real-world use ext3/ext4 in hours. The large scale deployments of ext3/ext4 dwarf the usage of ZFS.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#44

Post by mavhc »

antongribok wrote: post #16
tarruda wrote: post #14

Earlier quoted context omitted.

> Replication is with rsync and borg, not snapshots. I backup my files with borg, but I still use snapshots during the Borg backup to ensure files are not modified during the process

Yeah, of course you should be doing that. I was just trying to say that people shouldn't pretend like having snapshot-based, filesystem-level replication alone (which can be a very efficient way of replicating data changes) is a good backup strategy.

If you don't rebuild the zfs volume on the backup computer aren't you left with a) unlimited incremental backups, or b) doing a full backup every x days?

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#45

Post by lproven »

AndrewDavis wrote: post #30
antongribok wrote: post #25

Earlier quoted context omitted.

I guess I meant it as having a separate repair tool, but yes, you're of course correct. To me it was always weird to not have a separate tool for being able to do an offline ZFS repair. With regards to data corruption, I mean, this is exactly why I moved to btrfs, because it's able to catch bit rot same as ZFS. I think ultimately this is why ZFS data corruption bugs always are such a big deal... It's because in a lot…

Could you please explain what you mean by a separate repair tool kit?

I think he means an equivalent for `fsck`.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#46

Post by lproven »

worthless-trash wrote: post #43
buildbot wrote: post #39

Earlier quoted context omitted.

But it's totally different than corruption in other filesystems. People are acting like upgrading to ZFS 2.2.0 ate all their data like XFS used to back in the bad old days. I remember once the power went out at my house, and the entire XFS filesystem was irreparably damaged. This bug is super hard to trigger, and has roots back to 2006 and only surfaced with a recent coreutils update. Yet there are numerous posts in…

> But it's totally different than corruption in other > filesystems. People are acting like upgrading to ZFS 2.2.0 > ate all their data like XFS used to back in the bad old > days. I remember once the power went out at my house, and > the entire XFS filesystem was irreparably damaged. Different bugs manifest differently due to the coding, this is just the same. ZFS wouldn't even have a hundredth of the testing of rea…

ext4 released: 2008

https://archive.ph/20120529150649/http://git.kernel.org/?p=l...

ZFS released: 2005

https://web.archive.org/web/20130619165135/https://blogs.ora...

This is Linux local bias. There is more to Unix than Linux.

Back around 2001 when work on ZFS began, I very strongly suspect that there were several orders of magnitude more storage on ZFS on Solaris in datacentres around the world than there were on all Linux filesystems combined.

Just because you are more familiar with the FOSS tool doesn't mean that it's more mature or more widely used or more deployed in production.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#47

Post by db48x »

lproven wrote: post #45
AndrewDavis wrote: post #30

Earlier quoted context omitted.

Could you please explain what you mean by a separate repair tool kit?

I think he means an equivalent for `fsck`.

`zfs scrub` is equivalent to `fsck`.

It just has a different name, and it can fix more kinds of corruption (including corrupted data, which `fsck` for other file systems can never do), and can do so on line. Of course zfs can fix those same errors on line while doing ordinary reads, so really all `zfs scrub` does is read everything.

`zfs scrub` is better than `fsck`.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#48

Post by frafra »

moss2 wrote: post #42
buildbot wrote: post #10

Earlier quoted context omitted.

I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.

I'm unreasonably annoyed you used etcetera instead of et cetera . 'Et' is Latin for 'and'. The term means "and other things", not "andotherthings". No, I have nothing important to add to the ZFS discussion.

etcetera is found in ~500 years old books, and et cetera itself comes from et caetera, from the Greek καὶ τὰ ἕτερα. Which one should we use then? Should we only use the Latin from 1 BCE/CE or the vulgar Latin that evolved after? :-)

Maybe I do not feel it like a big issue, since many Latin languages use it a single word (such as Italian with eccetera, Spanish, Portuguese), due to the evolution of the language.

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#49

Post by db48x »

1letterunixname wrote: post #13

I come from the GPFS, Lustre, and Panasas world of HPCC. Personally, I used (past tense) ZoL in 2014-2017 on Ubuntu. The issue is that the array eventually entered an unrecoverable state where it could no longer be mounted RW. That wasn't the end of the world, but the support from ZoL was to shrug at it. That was the end of that because without support and without pride, something that appears shiny is effectively us…

If you had a Solaris support contract then they might have had an engineer poke around in your dead filesystem with `zdb` to see if they could salvage anything. `zdb` still exists, so you could have done that yourself. Why didn’t you?

Re: Tell HN: ZFS silent data corruption bugfix – my research results

#50

Post by cangeroo »

AndrewDavis wrote: post #7

This bug shouldn't really scare people. It's requires such an incredibly specific workload to hit Here's a post by RobN (the dev who wrote the fix) on the ZFS On Linux mailing list > There's a really important subtlety that a lot of people are missing in this. The bug is _not_ in reads. If you read data, its there. The bug is that sometimes, asking the filesystem "is there data here?" it says "no" when it should say…

> ask yourself if you've ever had highly parallel workloads that involve writing and seeking the same files at the same moment.

Uhhhh, databases?

Post reply on HN