21–30 of 130 posts
"Molex to SATA, lose all your data" was drilled into my head at an early job and appears to be what's depicted there.
If your disks are set up in a similar way, you might have a very immediate need for a good backup script :)
rsnapshot[1] is what I used on FreeBSD with a snapshot. It is like a well-tested version of the author's rsync and ssh blog post. I have a blog post here[2] describing my setup. It saved my bacon multiple times. Also, rsnapshot works in pull mode, so no client is needed on my Linux/macOS desktop or server except ssh and rsync. [1] https://github.com/rsnapshot/rsnapshot [2] https://www.cyberciti.biz/faq/howto-install-…
Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…
Can't help but throw a couple cents/pennies around whenever an "almost perfect" or "advanced" script makes my knee jerk so wildly. So here goes. For this part: pro_on=$(ps aux | grep -c rsync) # if someone is using rsync # grep is also producing one entry so -gt 1 `grep` can be excluded from the output by putting square brackets around one character of the search pattern, like so: pro_on=$(ps aux | grep -c rsyn[c]) A…
That's a great trick, thanks! I'm so used to adding '| grep -v grep' at the end of a 'ps | grep' command, your way is much nicer.
Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…
Make sure your borg repos are copying properly and in full! I had a horrible realisation that my borg backups were timing out on the offiste copy, meaning the resulting offsite backup I had was non-existent. The heartbreaking error message Inconsistency detected. Please run "borg check [repository]" - although likely this is "beyond repair" Course following the 3-2-1 rule you're probably good, but aye I'm treating bo…
A backup that isn't tested is not a true backup, it is a disappointment waiting to be found! This can happy with any backup tool, my hand-crafted¹ rsync based scripts included.
Testing isn't hard to setup if you don't mind the final step being manual. Snapshots have a checksum file, and daily one is picked and rechecked, any difference is an indication of bit-rot on the storage medium or something accidentally getting deleted/modified otherwise. After the newest copy is created by the main backup script a list of files not touched since the backup started is made and pushed up, the backup site checks those files and sends the result back so it can be compared. Any difference in these checksums results in an email to an account that makes my phone shout in a distressed manner. The manual part here is occasionally checking the results manually because not getting an email could either mean all is well or that something has broken to the point that the checks aren't running at all².
For specific systems like my mail server I have a replica VM, not visible to the Internet at large, that wipes itself and restores from the latest off-site backup. I look at that occasionally to see that it is running and has the messages I've recently sent and received. As a bonus this VM could quickly be made to be publicly available and take over with a few firewall DNS changes, should the main mail server or its host physically die, and even if it doesn't take over its existence proves that the restore method is reliable should I need to restore the one in the primary location. Some extra automated checks could be added to this too, but again there comes a point where writing the checks takes more time than just doing that manually³ and I'd still do it manually out of paranoia anyway.
[1] If I'm honest, “string together” would be much more accurate than “crafted”
[2] I could automate that a bit too, but then that automation still needs to be verified occasionally, it quickly gets to the point where it is double-checks all the way down and making sure you check manually occasionally is far far more maintainable a system.
[3] If these were a business thing rather than personal services, then the automated procedure vs manual checks desirability balance might change somewhat.
The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.
Bet yes pipefail and nounset should definitely be set.
Can't help but throw a couple cents/pennies around whenever an "almost perfect" or "advanced" script makes my knee jerk so wildly. So here goes. For this part: pro_on=$(ps aux | grep -c rsync) # if someone is using rsync # grep is also producing one entry so -gt 1 `grep` can be excluded from the output by putting square brackets around one character of the search pattern, like so: pro_on=$(ps aux | grep -c rsyn[c]) A…
> `grep` can be excluded from the output by putting square brackets around one character of the search pattern That's a great trick, thanks! I'm so used to adding '| grep -v grep' at the end of a 'ps | grep' command, your way is much nicer.
The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.
It makes sense to opt into errexit for select blocks/sections in scripts and under certain circumstances, but having it default-on is a recipe for quite a bit of head-scratching in the future.
This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises.
rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies.
Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most file metadata and they usually need many non-default flags to make complete file copies.
Nevertheless, rsync is the best of all alternatives, because when invoked correctly it accurately copies all file metadata even between different file systems and different operating systems, in cases when other copying utilities may lose some file metadata without giving any errors or warnings for the users.