Correct Backups Require Filesystem Snapshots
cyounkins.medium.com
Correct Backups Require Filesystem Snapshots
1–10 of 78 posts
Re: Correct Backups Require Filesystem Snapshots
#2Completely true
But also, if you have no backups, incorrect ragged backups of only non-changing files are a _hell_ of a lot better than no backups at all.
And if you get in the habit of doing incremental backups every day, you might be lucky enough to find a non-corrupted version of a file that's a few days stale.
Re: Correct Backups Require Filesystem Snapshots
#3Completely true But also, if you have no backups, incorrect ragged backups of only non-changing files are a _hell_ of a lot better than no backups at all. And if you get in the habit of doing incremental backups every day, you might be lucky enough to find a non-corrupted version of a file that's a few days stale.
For a long time I didn't know where to start with backups, because I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.)
Eventually I realized that some backups are better than no backups at all, and prioritized getting _something_ in place to at the very least have copies of important files copied regularly to another drive somewhere.
Re: Correct Backups Require Filesystem Snapshots
#4Completely true But also, if you have no backups, incorrect ragged backups of only non-changing files are a _hell_ of a lot better than no backups at all. And if you get in the habit of doing incremental backups every day, you might be lucky enough to find a non-corrupted version of a file that's a few days stale.
For a long time I didn't know where to start with backups, because I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.) Eventually I realized that some backups are better than no backups at all, and prioritized getting _something_ in place to at the very least have copies of important files copied regularly to another drive somewhere.
> I was convinced that incomplete or unreliable backups were a total waste of time and money. (Which tbf isn't 100% false.)
It depends a lot on your use case.
If you're backing up a production database system that's in constant use, then the files are almost certain to change while the backup is in progress. And a backup of everything on the database server except the data probably isn't what you want!
On the other hand, if I'm backing up my personal computer, my family photos aren't getting regularly overwritten - and if the backup of my web browser cache is inconsistent? So be it.
Re: Correct Backups Require Filesystem Snapshots
#5Actually, I have a hard time to understand this. While snapshot reduces file corruptions, it is not a guarantee the best I understand.
A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption).
Thus, any discussion on file corruptions without a file system that supports transaction, requires every application to use a competent database underneath (SQLite) at least.
And even with transactional support in a file system or using a database, you need every application to have the correct business logic that does the transaction correctly as well.
All-in-all, correct backup cannot be solved universally without knowing all the applications. The best we can do is to probabilistically avoid obvious issues, i.e. using FS-level snapshot.
Re: Correct Backups Require Filesystem Snapshots
#6Most of what is described there seems like "backup larping".
I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running.
Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at the time. When my laptop dies I probably can just remember what was needed to be done and redo the work.
Restoring file system snapshot would usually be much more hassle than filling in single invoice again or downloading it from some provider again.
For web applications there is usually SLA where they specify how much data can be lost like 1 hour or 30 mins - but no one will realistically guarantee "no data lost" - because imagine doing full snapshot of 1 Terabyte drive it takes I suppose at least an hour anyway.
Re: Correct Backups Require Filesystem Snapshots
#7Given ZFS it's potentially worth looking at https://github.com/jimsalterjrs/sanoid
Re: Correct Backups Require Filesystem Snapshots
#8https://www.arqbackup.com/ uses snapshots both on macOS and Windows!
Re: Correct Backups Require Filesystem Snapshots
#9Actually, I have a hard time to understand this. While snapshot reduces file corruptions, it is not a guarantee the best I understand. A corrupted file can manifest itself in many ways. But ultimately, it has to manifest itself as a business logic error, i.e. you increased balance on one entity but didn't on another, causing sum of balances to change (a corruption). Thus, any discussion on file corruptions without a…
Yes, you are correct in that it does not guarantee correctness. Point-in-time snapshots are necessary but not sufficient. Without them, the possible corruption scenarios are infinite and cannot be handled or even detected. With them, it is up to the applications to do the right thing in what is equivalent to a power loss event.
Re: Correct Backups Require Filesystem Snapshots
#10Most of what is described there seems like "backup larping". I don't need to backup my Discord and most likely I will be able to simply restore any iTerm configuration that I have in less time than setting up and keeping file system snapshots running. Any DEVONThink or my excel spreadsheets or invoices / documents that I care about I can simply backup after I am done working on them. I usually work on one or two at t…
Sure, I've been accused of being overly concerned with correctness. :-)
> because imagine doing full snapshot of 1 Terabyte drive it takes I suppose at least an hour anyway.
On copy-on-write filesystems, snapshots are nearly instantaneous.
> time than setting up and keeping file system snapshots running
As shown in the code snippet, pretty much all it takes is a few snapshot commands around the backup command and changing the source directory to the snapshot mount point.