Live data from Hacker News

ZFS on Linux: Unlistable and disappearing files

github.com

111–120 of 168 posts

Re: ZFS on Linux: Unlistable and disappearing files

#111
post #101

Earlier quoted context omitted.

> The actual ZFS on systems which have had ZFS for decades is not affected at all. ZFS hasn't existed "for decades".

It has existed since 2002, and was officially put back into onnv on 31.10.2005, and made available in Solaris 10 update 2 at the end of June 2006. So, it has existed for at least one decade and more.

It hasn't existed since 10 april 1998 therefore "decades" is inaccurate. Furthermore, since it wasn't publicly available from 10 april 2003 (who cares how long it was in development) if you round fairly, then its accurate to say "more than a decade" but "decades" is inaccurate. Because "decades" means "more than 1 decade". It denotes "at least 20 years".

That's regardless of the subject we discuss or how good it is etc.

Anyway, the original parent admitted their mistake (thank you) so I'm done with that discussion.

Re: ZFS on Linux: Unlistable and disappearing files

#112
post #44

Earlier quoted context omitted.

You are welcome. I should clarify the snapshots remark. The problem with how this interacts with snapshots is that the snapshots containing orphaned files cannot be repaired by software without BPR. They can only be listed by software for deletion by the administrator. Also, for the dataset’s tip, the future tool to repair it can only put the orphaned files into a lost+found directory without the original file names.

lost+found is an incorrect approach for ZFS. If you feel you have to regress to what was 40 years ago and was a bad solution, then the misfeature causing it needs to be engineered instead of hacked together, or better yet not implemented at all. If you have to break from mainstream ZFS and regress that violently, it’s the wrong approach.

Where do you suggest recovered file data (when metadata has been lost) should be put in the event of repair after a file system driver bug?

Re: ZFS on Linux: Unlistable and disappearing files

#113

Earlier quoted context omitted.

This is a good reminder for everyone that snapshots are not backups. Also, thank you for all of the hard work on ZoL!

“This is a good reminder for everyone that snapshots are not backups.” Were this not ZFS, I would agree with you, however, since zfs rollback will revert to previous state and if the snapshot is on a redundant vdev, there is no difference between that and losing your backup due to a damaged tape. In fact, zfs snapshots are exactly how time machine on illumos based operating systems is implemented.

Backups have three characteristics: redundancy, versioning and distribution. ZFS only fulfils two.

If ZFS is your backup target, then you're making more sense.

Re: ZFS on Linux: Unlistable and disappearing files

#114
post #25

Earlier quoted context omitted.

I have been running ZFS on macOS (OS X), illumos and Solaris for a good 7 years or so now. A major part of the reason I switched over fully, despite some warts, was that I experienced actual and significant data rot from stuff I was carrying forward under XFS (IRIX), HFS, etc. I don't consider my personal stuff to go back that long, but I still have things from 1993 or so that matter to me. I did a review around 2009…

I am still somewhat suprised there are still not relatively simple filesystems that don't do this. Make it the one feature it does and does properly. Adding enterprise features will just increase development time (and cost) and add bugs. Basically an online archival filesystem, that's how most people use their home computers. Requirements: 1. Checksumming on all files 2. Minimise assumptions of ram correctness 3. Dis…

>As much as I love subvolumes and snapshotting, I feel like CoW probably adds too much complexity to make it (easily) reliable.

Despite this bug, ZFS still seems reliable and does everything you ask. And any reliability you feel is gained by dropped CoW is offset by the reliability lost by being susceptible to power outages corrupting data.

Re: ZFS on Linux: Unlistable and disappearing files

#115
post #77

Earlier quoted context omitted.

i have been using btrfs in production for years now and it has never failed me, and i am doing hundreds of snapshots and send/receiving, i've reconfigured raids on the fly and went from 6 disk raid 10 of mixed size to raid 10 of same size. we have in some cases had many power failures with no data loss. We also have some set up using md raid and some using hardware raid... when i hear people dogging btrfs it just spe…

I use OpenSuse Tumbleweed with BTRFS on a laptop. This is nothing special - a 512GB SSD, no RAID, about 350GB of data and a bunch of snapshots. It was basically the out of the box configuration plus some snapper config to regularly snapshot /home. One day I shut it down and it wouldn’t boot - the BTRFS filesystem had gotten itself into a state where it would mount ok read-only but hang the system when mounting read/w…

Huh. I had that same problem on two machines running Tumbleweed a couple of months ago. In both cases, the only working approach I found was to wipe the root partition and install from scratch.

/home was in a separate partition, so it was not a tragedy, but annoying nevertheless. This has never happened to me before, unless the underlying hardware was about to retire.

Re: ZFS on Linux: Unlistable and disappearing files

#116
post #21

At least it's no btrfs. What a disaster that filesystem's been.

I've been running a two-disk RAID1 setup, alongside a single disk root drive (for snapshots) for almost two years without a single issue. I think there's still a lot of FUD being spread on account of the RAID 5/6 write hole still existing, for which this is the latest update:

> The write hole is the last missing part, preliminary patches have been posted but needed to be reworked.

Re: ZFS on Linux: Unlistable and disappearing files

#117

Earlier quoted context omitted.

This is a good reminder for everyone that snapshots are not backups. Also, thank you for all of the hard work on ZoL!

also a good time to remind people that backups are not backups unless they're geographically diverse. back up to an off site location in case your (home|office) burns down or suffers some other total catastrophe.

>also a good time to remind people that backups are not backups unless they're geographically diverse

I don't mean to jump on you here, but seriously I don't think this is fair, and additionally an attitude that I think can sometimes be at least mildly harmful. Data redundancy is a spectrum, not binary. Each additional option one might use helps against additional threat scenarios, but also at additional cost, complexity, and usage requirements. Some of the other replies to your comment are correct on things like "testing" and such, but I'd argue the ultimate Rule 0 of "backups are not backups if..." would be "backups are not backups if they're not actually used". All the possible media and geographic redundancy in the world, all the possible verification and regular testing, none of it matters if the result is too inconvenient or just plain too expensive for the operation/users to bother with. Remember the kinds of situation diversity people face; huge portions of the world, even including business operations in America, have poor net access period, and a much higher slice then that are dealing with highly asymmetrical links even if the download is ok. The kinds of natural disasters, crime threats, and so forth all vary from place to place also. And even if their data is reasonably valuable, for some places even just an extra few hundred or few thousand dollars/euros a year isn't nothing.

What is binary is that anything is better then zero. I have genuinely dealt with people, not just regular users but folks wearing IT hats in rural SOHO scenarios, who got discouraged by being told that they didn't have "real backups" (implying naturally that they didn't "really care" about their data) because they failed to check all the boxes an enterprise in SV easily could, and ended up just kind of giving up on most anything. In my experience any sort of orderly backup process at all still isn't always the rule, so I worry about mental and implementation friction there, even though it's true there's a need to push on folks a bit to have at least a minimal quality solution too.

Still, if someone regularly plugs in a USB drive to their desktop and runs an rsync script or TM or something every evening when closing up? Yeah, that's a backup. No it doesn't cover the place burning down, but it does cover some primary hardware failure, users accidentally deleting something, possibly some ransomware (if used with care), etc. Maybe they have 2 drives, and stick one in a firesafe in a sealed bag, now maybe they've got a bit of protection from certain fires too if they're caught fast enough. Maybe they add on a bit of light net backup, just of key low size accounting documents and the like, what their 5/1 ADSL link can reasonably handle, ok that's better still. Etc. If they've got the discipline (and money) to have a bunch of drives or get a tape system and then regularly rotate a week or month's worth into a safe deposit box at their bank or something? Great. But there is no set number of 9s that needs to be hit before it's a real backup. 90% is real, and really better then 0%, even if it's worse then 99% or 99.9[99]%. Get in the habit, get in the habit of yearly reviews too at least, then keep improving as importance, opportunity and budget allows.

Re: ZFS on Linux: Unlistable and disappearing files

#119
post #115

Earlier quoted context omitted.

I use OpenSuse Tumbleweed with BTRFS on a laptop. This is nothing special - a 512GB SSD, no RAID, about 350GB of data and a bunch of snapshots. It was basically the out of the box configuration plus some snapper config to regularly snapshot /home. One day I shut it down and it wouldn’t boot - the BTRFS filesystem had gotten itself into a state where it would mount ok read-only but hang the system when mounting read/w…

Huh. I had that same problem on two machines running Tumbleweed a couple of months ago. In both cases, the only working approach I found was to wipe the root partition and install from scratch. /home was in a separate partition, so it was not a tragedy, but annoying nevertheless. This has never happened to me before, unless the underlying hardware was about to retire.

I wasn't so lucky since I had everything as one big BTRFS filesystem.

I did recover it by booting the installer in recovery mode, mounting the fs read-only, and doing a backup. Then I blew away the partition table and reinstalled.

I also learned a lesson about backups, mainly that I should have them.

Re: ZFS on Linux: Unlistable and disappearing files

#120
post #25

Earlier quoted context omitted.

I have been running ZFS on macOS (OS X), illumos and Solaris for a good 7 years or so now. A major part of the reason I switched over fully, despite some warts, was that I experienced actual and significant data rot from stuff I was carrying forward under XFS (IRIX), HFS, etc. I don't consider my personal stuff to go back that long, but I still have things from 1993 or so that matter to me. I did a review around 2009…

I am still somewhat suprised there are still not relatively simple filesystems that don't do this. Make it the one feature it does and does properly. Adding enterprise features will just increase development time (and cost) and add bugs. Basically an online archival filesystem, that's how most people use their home computers. Requirements: 1. Checksumming on all files 2. Minimise assumptions of ram correctness 3. Dis…

Checksumming is like a myth on HN already. No, it's not useful on its own to the user or to the filesystem, but it is useful to the filesystem if it can transparently heal the data. Whole disk replication doesn't address your reliability concerns as much as you think either, doesn't work all that well in general and is a burden to maintain. Doing it properly requires treating the filesystem as a distributed system where disks can join and leave, where everything is sharded, where rebalancing, self-healing, syncing, resyncing is all automatic and blazing fast. And so on.

Nowadays most of the work in storage systems is in distributed systems, and local filesystems are treated as just another unreliable layer.

Hope this gives you some ideas why there are no filesystems like that.

Post reply on HN