The wrong way to switch operating systems on your server

Anything you do in production should always be reversible at least one step back with a single command. You should always be able to roll back to the last-known-working version at the first sign of trouble with the new version. (Not that you should necessarily respond to all trouble with a roll back but you should always design your processes to keep this as one of your options.)

The way I usually do this is that I have a production PUSH and ROLLBACK script. There is a single symlink on the production server that points to the current working version of the code. PUSH makes a complete local copy of the current working version, changes the symlink to point to that, then pushes the new version, then changes the symlink to point to that. ROLLBACK just changes the symlink back to the backup copy. This is robust and easy to tweak to allow you to roll back as many levels as you like, though I've never had to go back more than one. If a problem doesn't manifest itself immediately you probably want to fix it by going forward, not back.

I'd even go a step further. It's not a backup until you've restored it using different hardware. You really want to know that your tape (or whatever) can be read by a different tape reader than the one that wrote it.

For me: rsync > tarsnap

I have a backup rsync script that parses a file I have that lists every path I want backed up. Yes, this considers dotfiles, so the poster's .env file would have been backed up. My script runs locally, backs up to my main (home) server, and then does another rsync to a 'cloud' server. Want to backup a new file or path? Add it to the manifest file. Adding another server or device? Build another manifest script, have rsync write to the same dir on the server, it'll automatically get sync'd to the cloud server too.

Maybe, it depends on your data-loss and time to restore flexibility. Test regularly enough that you are confident it works. If you don't test at all you can't be confident that it works at all.

For instance, if you have a full/diff/log backup cycle for a database for instance, perhaps test restore each full backup. Just be aware that if a full backup happens daily your comfort zone for data loss in the case of a disaster needs to be at least a day worth of work.

Also, if you backup a range of data and your method allows for partial restores, you might do partial restores of key information far more often than you test everything. You could spread the testing load temporaly: check the really import parts every time and cycle through the other bits less often.

Also if your concern is man-time, automate the process as much as possible. My mail server (running Zimbra) has a small replica in a VM that restores itself from the latest backup once a day and sends me a message to say what the last message in its queue was. This way if I don't get the message, or it says the last message was too far ago, I know something has gone wrong and the backup or the restore failed. I manually log in regularly to inspect a little deeper too, this is slow as the VM has far lower resources than the main box. If properly separated from other resources (so there are no single points of failure that can take it out along with everything else - mine isn't currently that well arranged) then this copy becomes an extra secondary single-snapshot backup itself.