Making Backup Validation Easier
brokensandals.net
Making Backup Validation Easier
1–10 of 18 posts
Re: Making Backup Validation Easier
#2This seems like a worse than just hashing the file. Random bit flips will probably go undetected using this method, but won't be with hashing.
Re: Making Backup Validation Easier
#3This seems like a worse than just hashing the file. Random bit flips will probably go undetected using this method, but won't be with hashing.
The contents of my backups are never the same, not from one single day to the other - so hashes would be useless.
Re: Making Backup Validation Easier
#4This seems like a worse than just hashing the file. Random bit flips will probably go undetected using this method, but won't be with hashing.
The contents of my backups are never the same, not from one single day to the other - so hashes would be useless.
You don't need hashes to match between days at all though. You simply hash the file that was just backed up, and the the backup copy of it, then compare the two
Re: Making Backup Validation Easier
#5I think making a list of the files to be copied and their hashes, then a list of files that were copied and their hashes, then comparing the 2 lists should provide an even quicker way to validate. Or even hashing the entire source and destination (hash of the list of hashes) and providing both values to the user to visually compare.
As far as I can tell the method described in the article doesn't really validate the backups in any way, just provides some statistics that will fail in very plausible ways.
And of course, if the data is important to you and there are special circumstances that could affect the process, nothing beats an actual restore test.
Re: Making Backup Validation Easier
#6Earlier quoted context omitted.
The contents of my backups are never the same, not from one single day to the other - so hashes would be useless.
You don't need hashes to match between days at all though. You simply hash the file that was just backed up, and the the backup copy of it, then compare the two
This works except in the case where your backups include live database files (where you put the database in extended logging mode, back up the data files while they are being modified, then back up the logs).
I haven't found a good way to verify these without doing a full database restore and seeing if the logs apply cleanly, along with having the DB do internal checks.
Re: Making Backup Validation Easier
#7Earlier quoted context omitted.
The contents of my backups are never the same, not from one single day to the other - so hashes would be useless.
You don't need hashes to match between days at all though. You simply hash the file that was just backed up, and the the backup copy of it, then compare the two
I guess it depends somewhat on what you're backing up and what the anticipated failure modes might be. As an example, if there was a bug in my todo software that deleted a bunch of entries, the hash scheme wouldn't pick that up. You've just successfully backed up corrupted data, and you're not aware of it. SQL dumps would be another good example of this. If one day you do a backup and the backup reports that it has archived significantly fewer rows than yesterday, you know something's up. Maybe a fault lost some data, maybe the archiver is broken, etc.
Re: Making Backup Validation Easier
#8What I did is to do all my work in a VMware virtual machine.
Then I wrote software for backing up VM's automatically (disclaimer: this is a commercial product I sell)
There's options for getting an email on success, failure or both. The VM files are all hashed.
VMs are easy to restore, so an actual restore is pretty easy without risking to overwrite the original. If a file hash does not match on restore, then my software will complain, but continue the restore anyways.
FWIW, all my code etc... is also in source control, so I am not relying on a single layer for that.
Re: Making Backup Validation Easier
#9Earlier quoted context omitted.
You don't need hashes to match between days at all though. You simply hash the file that was just backed up, and the the backup copy of it, then compare the two
This works except in the case where your backups include live database files (where you put the database in extended logging mode, back up the data files while they are being modified, then back up the logs). I haven't found a good way to verify these without doing a full database restore and seeing if the logs apply cleanly, along with having the DB do internal checks.
Isn't this use case solved by snapshotting the volume, then backing up the snapshot? Since the snapshot captures the filesystem state at a point in time, any database that's crash-tolerant should be fine. Snapshotting is natively supported on Windows and Macs, not sure about linux.
Re: Making Backup Validation Easier
#10Earlier quoted context omitted.
You don't need hashes to match between days at all though. You simply hash the file that was just backed up, and the the backup copy of it, then compare the two
I guess it depends somewhat on what you're backing up and what the anticipated failure modes might be. As an example, if there was a bug in my todo software that deleted a bunch of entries, the hash scheme wouldn't pick that up. You've just successfully backed up corrupted data, and you're not aware of it. SQL dumps would be another good example of this. If one day you do a backup and the backup reports that it has a…
What you're describing is significantly harder than what's described in the blog post. Not only do you have to validate a file looks like a .jpg/.json/.zip file, you also need to validate that it looks semantically correct (ie. the file format is valid but a chunk of it is missing).
Most people solve this issue by keeping multiple versions, not by trying to "validate" the backups somehow.