Earlier quoted context omitted.
It really is better. The key difference to understand is that in ext4 any form of metadata corruption which can be automatically fixed requires you to take the filesystem off–line (to unmount it), and then run fsck. Meanwhile with zfs, any form of metadata corruption which can automatically be fixed is simply fixed right on the spot, transparently. In truth fsck is a wart, a kludge, a bag on the file system design.
I am passingly acquainted with ZFS. The key thing to realize is that you can't, actually, automatically fix every problem, sometimes you have found a logic problem which results in an impossible outcome and you need someone to manually clean it up. In a world without flaws, it would be great to never need that. But the thing about theory and practice is that in theory, they never differ, but in practice...
Tell HN: ZFS silent data corruption bugfix – my research results
71–80 of 90 posts
Re: Tell HN: ZFS silent data corruption bugfix – my research results
#72Earlier quoted context omitted.
I am passingly acquainted with ZFS. The key thing to realize is that you can't, actually, automatically fix every problem, sometimes you have found a logic problem which results in an impossible outcome and you need someone to manually clean it up. In a world without flaws, it would be great to never need that. But the thing about theory and practice is that in theory, they never differ, but in practice...
I know that. But fsck only fixes the ones that it can fix automatically. With anything else, you are completely on your own. Get a hex editor and go to town. With zfs, if there is some kind of problem that cannot be automatically fixed then at least you have one more tool available: zdb. It’s a _debugger_ for zfs filesystems. It will show you everything, more than you ever wanted to know. It is way better for fixing…
Re: Tell HN: ZFS silent data corruption bugfix – my research results
#73https://github.com/openzfs/zfs/pull/15529#pullrequestreview-... Honestly, ZFS is the best thing on the (Free)BSDs only... On Linux it doesn't even use the page cache, and you conflict severely with L2ARC. I know there's a variety of people who don't care, but still for real users it's not an actual option.
For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me. I finally moved to btrfs earlier this year, and so far I'm glad I did. I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Ta…
Re: Tell HN: ZFS silent data corruption bugfix – my research results
#74Earlier quoted context omitted.
I feel like there has been kind of a weird concerted effort to push that zfs is bad due to this bug and how trust has been lost etcetera etcetera - super annoying when most other filesystems just corrupt your data and nobody will ever know it happened. I’ve experienced bad data corruption on xfs, btrfs, ext2, and ext4. So far zfs is been nothing but perfect.
No, you're just hearing about ZFS corruption like you've heard corruption in other filesystems, this wont be the first and last. Its easy to feel like its being targeted with some kind of campaign, however the truth is rarely that exciting. The software is being used by more people, which means it will expose more bugs. No software is perfect, it just now starting to be abused enough to be important enough to be talk…
I don't think I ever heard about this, apart from in the context of the ZFS bug. And although inline_data is niche, ext4 as a whole I would argue is not.
Actually lseek seems to have been broken on most filesystems at some point: https://bugs.gentoo.org/891125 https://github.com/gluster/glusterfs/issues/894
And apparently apart from modern coreutils using that, it is mostly gentoo users hitting the bugs in lseek.
Re: Tell HN: ZFS silent data corruption bugfix – my research results
#75Earlier quoted context omitted.
I know that. But fsck only fixes the ones that it can fix automatically. With anything else, you are completely on your own. Get a hex editor and go to town. With zfs, if there is some kind of problem that cannot be automatically fixed then at least you have one more tool available: zdb. It’s a _debugger_ for zfs filesystems. It will show you everything, more than you ever wanted to know. It is way better for fixing…
zdb is read-only. It's not fixing anything, just telling you what's going on.
Re: Tell HN: ZFS silent data corruption bugfix – my research results
#76Earlier quoted context omitted.
zdb is read-only. It's not fixing anything, just telling you what's going on.
Don’t be an idiot. You can fix more with zdb and a hex editor than you can with the hex editor alone.
The discussion was about the need for tools to make it easier to handle cases where you couldn't automatically handle repairing them, and your statement was that zdb is very useful, which is true, but it doesn't fix anything.
fsck for various filesystems has a bunch of common cases like "this is an orphaned file, should I save it or mark it free?", and something similar would indeed be useful for a number of failure cases in ZFS which require more explicit instruction on what to do about it because you can't easily automatically resolve it.
`zpool scrub` is very useful, but ZFS could still benefit from automated tooling to handle some common failure modes, not just let you write bespoke tooling every time.
Re: Tell HN: ZFS silent data corruption bugfix – my research results
#77Earlier quoted context omitted.
Don’t be an idiot. You can fix more with zdb and a hex editor than you can with the hex editor alone.
That seems rather rude. The discussion was about the need for tools to make it easier to handle cases where you couldn't automatically handle repairing them, and your statement was that zdb is very useful, which is true, but it doesn't fix anything. fsck for various filesystems has a bunch of common cases like "this is an orphaned file, should I save it or mark it free?", and something similar would indeed be useful…
This is an example of an automated fix that zfs just handles transparently, without needing to prompt the user. Or it would, if zfs could even have orphaned files, which it cannot.
I don’t know why you think this is such a win for fsck, which doesn’t even bother to give you any idea what the file was. It doesn’t try to show you the contents, and it probably doesn’t know what the file was called, or why it was deleted. Or even if it _was_ deleted; a file could be orphaned merely because the data was written but the write to the directory entry got lost. The user has nothing to go on and just guesses, or says `y` for everything. Useless.
> `zpool scrub` is very useful, but ZFS could still benefit from automated tooling to handle some common failure modes
This is precisely and exactly what scrubbing does! All failure modes that can be automatically fixed, whether they are common or not, are transparently fixed without even needing to unmount anything.
zdb is there for the really rare cases where there is so much damage to the filesystem that zfs cannot even mount it safely. Other filesystems don’t have anything like it.
Re: Tell HN: ZFS silent data corruption bugfix – my research results
#78Earlier quoted context omitted.
> But it's totally different than corruption in other > filesystems. People are acting like upgrading to ZFS 2.2.0 > ate all their data like XFS used to back in the bad old > days. I remember once the power went out at my house, and > the entire XFS filesystem was irreparably damaged. Different bugs manifest differently due to the coding, this is just the same. ZFS wouldn't even have a hundredth of the testing of rea…
ZFS and ext4 is so completely different on a fundamental level that there is zero reason comparing them, it just shows your "expertise" on the topic..
Re: Tell HN: ZFS silent data corruption bugfix – my research results
#79Earlier quoted context omitted.
> But it's totally different than corruption in other > filesystems. People are acting like upgrading to ZFS 2.2.0 > ate all their data like XFS used to back in the bad old > days. I remember once the power went out at my house, and > the entire XFS filesystem was irreparably damaged. Different bugs manifest differently due to the coding, this is just the same. ZFS wouldn't even have a hundredth of the testing of rea…
ext4 released: 2008 https://archive.ph/20120529150649/http://git.kernel.org/?p=l... ZFS released: 2005 https://web.archive.org/web/20130619165135/https://blogs.ora... This is Linux local bias. There is more to Unix than Linux. Back around 2001 when work on ZFS began, I very strongly suspect that there were several orders of magnitude more storage on ZFS on Solaris in datacentres around the world than there were on al…
Re: Tell HN: ZFS silent data corruption bugfix – my research results
#80Earlier quoted context omitted.
That seems rather rude. The discussion was about the need for tools to make it easier to handle cases where you couldn't automatically handle repairing them, and your statement was that zdb is very useful, which is true, but it doesn't fix anything. fsck for various filesystems has a bunch of common cases like "this is an orphaned file, should I save it or mark it free?", and something similar would indeed be useful…
> fsck for various filesystems has a bunch of common cases like "this is an orphaned file, should I save it or mark it free?" This is an example of an automated fix that zfs just handles transparently, without needing to prompt the user. Or it would, if zfs could even have orphaned files, which it cannot. I don’t know why you think this is such a win for fsck, which doesn’t even bother to give you any idea what the f…
If you use the zfs_recover parameter, then in a couple of cases, it will just permanently mark space as unallocatable forever because it can't figure out what owns it due to some errors, and you decided that was a better outcome than whatever error it was encountering. (That's what the "leaked" zpool property counts.)
Conceivably, you could write something to carefully figure out what, if anything, owns it, and allow it to be freed, or grab the contents of that region and drop it somewhere and then throw it out after the same safeguards, but by definition if you triggered that handling, something has gone wrong and you don't have a better automated intervention.
I wasn't arguing that ZFS has a case requiring orphaned files, but my point was that that was an example of "I don't know what to do with this, but I know enough to realize I can't decide what to do about this or just throw it out, so here, you do it."
Or if you, for example, wanted to do a destructive rewind on a pool because of some horrible edge case, then you might want some automated way to extract everything you're about to throw out, and that might require more work than just `zdb -R` if the pool won't import in the first place.
One example of something that would be useful to be able to do, particularly offline, would be when pools have some issue like spacemap corruption, you could conceivably walk the non-spacemap metadata to re-synthesize the spacemaps from whole cloth and write them out again. (And it's not an all-or-nothing thing, a number of versions many years ago would have very minor errors in how they computed spacemaps, which just mean zdb whines at you if you ask it to verify them even when the pool is offline, but don't interfere with the running otherwise.)
Could you convince the kernel code to do that for you on import if it's blocking import? Probably, though you'd probably want some out of band communication method to force it sometimes because I would bet there's any number of ways it might go awry and get too far into the woods of thinking it's "fine" before tripping some assertion.
Is it going to be faster to iterate on it from userland, particularly if you can simulate whether the import works from something like zdb with those conceptual spacemaps written out somewhere, versus rebooting on kernel panic? Absolutely.
But that's an example of a case you might have something so disgruntled it could be nice to reconstruct it outside of the normal import flow.
> All failure modes that can be automatically fixed, whether they are common or not, are transparently fixed without even needing to unmount anything.
`zpool scrub` doesn't fix anything except checksum errors. That's it. Any other class of flaw, it doesn't handle. I don't know why you think it does anything else, but I promise you, it absolutely does not.