An almost perfect rsync over SSH backup script • Hacker News Board

Re: An almost perfect rsync over SSH backup script

#11

Post by corobo » Fri, Feb 25, 2022, 1:47 PM UTC

throwaway81523 wrote: post #3

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

Make sure your borg repos are copying properly and in full!

I had a horrible realisation that my borg backups were timing out on the offiste copy, meaning the resulting offsite backup I had was non-existent. The heartbreaking error message Inconsistency detected. Please run "borg check [repository]" - although likely this is "beyond repair"

Course following the 3-2-1 rule you're probably good, but aye I'm treating borg repos as delicately as I would a striped raid now. I am also monitoring that `borg check` actually comes back successfully before considering the backup complete too :)

Also if you're in a small company, start up, etc.. Set yourself a 3-2-1+ backup system. + is the miracle backup that makes you look good.

Re: An almost perfect rsync over SSH backup script

#12

Post by DarylZero » Fri, Feb 25, 2022, 1:48 PM UTC

I love the irony of a document about a "perfect" script with a grammatical error in the very first word of the title.

Re: An almost perfect rsync over SSH backup script

#13

Post by klysm » Fri, Feb 25, 2022, 1:59 PM UTC

corobo wrote: post #11

throwaway81523 wrote: post #3

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

Make sure your borg repos are copying properly and in full! I had a horrible realisation that my borg backups were timing out on the offiste copy, meaning the resulting offsite backup I had was non-existent. The heartbreaking error message Inconsistency detected. Please run "borg check [repository]" - although likely this is "beyond repair" Course following the 3-2-1 rule you're probably good, but aye I'm treating bo…

This is why you gotta test your backups too

Re: An almost perfect rsync over SSH backup script

#14

Post by koolba » Fri, Feb 25, 2022, 2:11 PM UTC

The first two lines of the script are already wrong;

    #!/bin/bash

Should be:

    #!/usr/bin/env bash
    set -euo pipefail

That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

Re: An almost perfect rsync over SSH backup script

#15

Post by BeefWellington » Fri, Feb 25, 2022, 2:13 PM UTC

I wish more of these backup solutions (even simple scripts like this) would include zero trust features.

I'd like to have a backup system that I didn't just kludge together myself over a weekend that accomplishes both delta change detection and rotation as well as allows me to keep my data encrypted with a key only I control.

Re: An almost perfect rsync over SSH backup script

#16

Post by BeefWellington » Fri, Feb 25, 2022, 2:15 PM UTC

rurban wrote: post #2

Perfect? Ha. Wrong ps grep usage. MacOS specific. No shell quoting. At least he doesn't advocate rm -rf / by an incorrect usage of --delete.

> Perfect? Ha.

The cynic in me believes this was intentional to drive engagement and discussion.

It's that old adage of "if you want to learn something just say something wrong about it on the Internet" at work.

Re: An almost perfect rsync over SSH backup script

#17

Post by kzrdude » Fri, Feb 25, 2022, 2:21 PM UTC

I use duplicity. Far from perfect, but I hope it's better than rsync and similar.

Re: An almost perfect rsync over SSH backup script

#18

Post by hkt » Fri, Feb 25, 2022, 2:29 PM UTC

Honestly, I just use duplicity. It is easy to test backups and has a nice GUI if you're that way inclined. I'm too old for bash scripts when the stakes are a lifetime of memories or critical data at work.

Re: An almost perfect rsync over SSH backup script

#19

Post by darrenf » Fri, Feb 25, 2022, 2:33 PM UTC

Can't help but throw a couple cents/pennies around whenever an "almost perfect" or "advanced" script makes my knee jerk so wildly. So here goes. For this part:

    pro_on=$(ps aux | grep -c rsync)

    # if someone is using rsync
    # grep is also producing one entry so -gt 1

`grep` can be excluded from the output by putting square brackets around one character of the search pattern, like so:

    pro_on=$(ps aux | grep -c rsyn[c])

Also, this:

    minutes=$(($minutes - 1))

Can be more easily written as:

    ((minutes--))

Re: An almost perfect rsync over SSH backup script

#20

Post by thg » Fri, Feb 25, 2022, 2:33 PM UTC

My go-to backup solution, that can also manage rsync over ssh (among plenty other things), is synbak[0]. Short wrapper script to automatically mount the backup medium and it's really simple to (automatically) run a backup job. I use encrypted (LUKS) USB drives for that at home. Highly recommend it.

For encrypted backups to e.g. a NAS, Duplicity[1] is my go-to choice (full backup every month or so, with incremental backup every day inbetween).

[0]: https://github.com/ugoviti/synbak

[1]: https://duplicity.gitlab.io/duplicity-web/