51–60 of 78 posts
I've not been seeing any of this much vaunted reliability. 3 out of 3 of the last drives I lost data on were ZFS. Meanwhile NTFS and EXT4 has been fairly trouble free.
Though to be fair hard to tell how much of that is TrueNAS vs ZFS.
>On Linux I strongly recommend ZFS due to its long history of reliability I've not been seeing any of this much vaunted reliability. 3 out of 3 of the last drives I lost data on were ZFS. Meanwhile NTFS and EXT4 has been fairly trouble free. Though to be fair hard to tell how much of that is TrueNAS vs ZFS.
Earlier quoted context omitted.
I think it's not just between files, but within a file. The backup process opens some N byte file and starts copying it. Some portion of the file is backed up until byte K. At that point the application writes a transaction whereby some of the data is written above K, and some below K. The backup continues and backs up half the transaction above K, combining that with the old data below K before the transaction happe…
You write "transaction", but Unix filesystems don't know anything about any transactions from the application's point of view. To them it's just a bunch of write(2) calls. How is a ZFS snapshot going to make sure that it snapshots the state either before or after what an application considers a "transaction", but implements with a sequence of write(2), fsync(2) etc. calls? (EDIT: The rest of this paragraph is rescind…
The serial copy of the file does not have that property. We may be able to find some k such that the backup contains some writes newer than k, while missing some older ones.
The snapshot situation looks, to the application, like a crash and reboot, which it knows how to deal with (with test coverage and everything). The application recognizes that there is an uncommitted transaction and rolls things to the prior state. Or else sees that there is enough info to finish the transaction; just the commit part wasn't done.
The missing old writes situation doesn't seem easily recoverable; I have no idea how you would code defensively against that, other than assuming that the backup copy is made serially, and make sure that in every transaction, the writes occur in increasing offset order: the same direction. If the backup and transaction race in the same direction, then it's impossible for the backup to have "holes" whereby some older writes are missing.
Earlier quoted context omitted.
> I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.) It depends a lot on your use case. If you're backing up a production database system that's in constant use, then the files are almost certain to change while the backup is in progress. And a backup of everything on the database server except the data probably isn't what you want! On the other h…
It's been a while, but with e.g. Oracle 10, it was specifically stated in the docs that backing up the data-file(s), followed by the archive-logs, you would end up with a recoverable backup. I'm not sure if other databases are designed that way.
Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at t…
Easy when you work on simple documents. I wouldn't want my accountant doing that.
Earlier quoted context omitted.
You write "transaction", but Unix filesystems don't know anything about any transactions from the application's point of view. To them it's just a bunch of write(2) calls. How is a ZFS snapshot going to make sure that it snapshots the state either before or after what an application considers a "transaction", but implements with a sequence of write(2), fsync(2) etc. calls? (EDIT: The rest of this paragraph is rescind…
What ZFS will do it not capture a state before or after the transaction, but it will copy a state that the file actually has at some point during the transaction. The snapshot will have the state of the file which contains write_0, write_1, ...., write_k; and does not contain the effects of write_k+1, write_k+2, ... The serial copy of the file does not have that property. We may be able to find some k such that the b…
Earlier quoted context omitted.
I think it's not just between files, but within a file. The backup process opens some N byte file and starts copying it. Some portion of the file is backed up until byte K. At that point the application writes a transaction whereby some of the data is written above K, and some below K. The backup continues and backs up half the transaction above K, combining that with the old data below K before the transaction happe…
You write "transaction", but Unix filesystems don't know anything about any transactions from the application's point of view. To them it's just a bunch of write(2) calls. How is a ZFS snapshot going to make sure that it snapshots the state either before or after what an application considers a "transaction", but implements with a sequence of write(2), fsync(2) etc. calls? (EDIT: The rest of this paragraph is rescind…
The point of the article is that, after a restore from a backup that used a snapshot, whatever ends up on disk, it’s something that the application’s developers could have planned for.
On the other hand, if the backup didn’t use a snapshot, the application may be given something that those writing the application couldn’t have foreseen.
I guess not many applications will handle either case perfectly or even decently (who even tests for the related issue of disk full errors on save nowadays?), but databases should.
I also would expect/hope lots of embedded software with a file system to handle this well, not because they get restored from faulty backups much, but because they have to be able to recover from unexpected reboots.
even with file system snapshots, your backup can be corrupt. the example of the browser profile is a classic case. imagine multiple writes occur and all of them have to occur for the profile to be correct (either multiple files are being written or firefox will write multiple blocks to the file). if the snapshot operation occurs in the middle, then the snapshot will be "corrupt". I believe this is actually the entire…
> I believe this is actually the entire point of Windows' volume shadow service (which is sort of poo-pooed in the article), to enable applications to tell the snapshot mechanism "wait, I'm in the middle of a file system transaction" and then to pause writes until the snapshot operation occurs after they finish the in process transaction. Former maintainer of VSS here. Yes, that's exactly right. In fact, filesystem s…
>On Linux I strongly recommend ZFS due to its long history of reliability I've not been seeing any of this much vaunted reliability. 3 out of 3 of the last drives I lost data on were ZFS. Meanwhile NTFS and EXT4 has been fairly trouble free. Though to be fair hard to tell how much of that is TrueNAS vs ZFS.
This seems very unlikely without a lot of contributing factors. Unless you weren't running any RAID in which case ZFS didn't lose you data but it would've been quite upfront about corruption when it happened.
That's what I thought given that it does have a good rep.
Haven't given up on it yet, but definitely less trusting - 3 different drives in 3 different ways wasn't expected.
That said - all consumer class gear in janky environments so there may very well be other factors lurking.