An almost perfect rsync over SSH backup script • Hacker News Board

Re: An almost perfect rsync over SSH backup script

#51

Post by rsync » Fri, Feb 25, 2022, 4:53 PM UTC

throwaway81523 wrote: post #3

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

"Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it."

I'm still not clear - does the Hetzner storage box have the borg binary installed on their end ? As in, one could run:

  ssh user@hetzner borg --version

... or are you accessing borg over an sshfs mount, etc. ?

Asking for a friend ...

Re: An almost perfect rsync over SSH backup script

#52

Post by chasil » Fri, Feb 25, 2022, 4:55 PM UTC

koolba wrote: post #14

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

If I may add a few critiques...

  # Make sure no one else is using rsync
  pro_on=$(ps aux | grep -c rsync)

A better way to do that is with the flock utility.

  (
    flock -n 9 || exit 1 # Critical section-allow only one process.

    ...single thread shell script

  ) 9> ~/.empty_lock_file

Note that the flock utility is specific to Linux, but POSIX mkdir() is atomic and could be more portable.

  "${SOURCES[@]}"

POSIX shells do not support arrays. Iterating with read over a here document is more portable.

  minutes=$(($minutes - 1))

POSIX is specific that the $ prefix on a variable name can be omitted in an arithmetic expression.

  ECHO="/bin/echo"

Many shell scripts never use echo, and this is a good idea. 'NEVER use echo like this. According to POSIX, echo has unspecified behavior if any of its arguments contain "\" or if its first argument is "-n".' http://www.etalabs.net/sh_tricks.html

Perhaps use this instead, in a subshell to avoid stomping on variables:

  myecho () ( z=''; for x; do printf "$z%s" "$x"; z=' '; done; )

Re: An almost perfect rsync over SSH backup script

#53

Post by aedocw » Fri, Feb 25, 2022, 4:58 PM UTC

throwaway81523 wrote: post #3

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

Fully agree, Borg is awesome. I wrote up how I'm using it here[1]. In short, borg backup to a local machine, and that machine uses rclone to copy the backups to an S3 bucket off-site. I've had multiple occasions to restore stuff successfully, and I never have to think about whether my data is retrievable.

[1]https://opensource.com/article/17/10/backing-your-machines-b...

Re: An almost perfect rsync over SSH backup script

#54

Post by zeec123 » Fri, Feb 25, 2022, 5:13 PM UTC

senko wrote: post #42

koolba wrote: post #14

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

Agreed on fail on error, but the first one is needlessly pedantic. Find me a single Linux distro where bash, if installed, is not available in /bin

NixOS

Re: An almost perfect rsync over SSH backup script

#55

Post by clagio » Fri, Feb 25, 2022, 5:37 PM UTC

rsync wrote: post #51

throwaway81523 wrote: post #3

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

"Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it." I'm still not clear - does the Hetzner storage box have the borg binary installed on their end ? As in, one could run: ssh user@hetzner borg --version ... or are you accessing borg over an sshfs mount, etc. ? Asking for a friend ...

The First. Hetzner officially support Borg on their storage box

Re: An almost perfect rsync over SSH backup script

#56

Post by justin_oaks » Fri, Feb 25, 2022, 5:40 PM UTC

I use Restic (https://restic.net/) for backups. Encrypted backup, incremental, deduplicating, and supports many storage backends.

Most recent discussion on Hacker News: https://news.ycombinator.com/item?id=29209455

Re: An almost perfect rsync over SSH backup script

#57

Post by 0xbadcafebee » Fri, Feb 25, 2022, 5:56 PM UTC

> # avoidng collisions with other rsync processes

Use https://github.com/instacart/ohmycron

> MONTHROTATE=monthrotate # use DD instead of YYMMDD

Use https://rotate-backups.readthedocs.io/en/latest/readme.html

> $RSYNC -avR "$SOURCE" "${RSYNCCONF[@]}"

Create a one-command script with the hardcoded rsync command you want to use and replace the directory to sync as a command-line argument, e.g.

  #!/bin/sh
  rsync -avR \
    --delete \ 
    --exclude=/Volumes/Raid/.DocumentRevisions-V100 \
    --exclude=/Volumes/Raid/.TemporaryItems --exclude=/Volumes/Raid/.Trashes \
    --exclude=/Volumes/Raid/.apdisc \
    "$@"

> $DATE >> $LOG

Use logrotate

> mail a report

Use Cron's built-in mail feature

Re: An almost perfect rsync over SSH backup script

#58

Post by 0xbadcafebee » Fri, Feb 25, 2022, 6:08 PM UTC

koolba wrote: post #14

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

#!/usr/bin/env bash and set -u are always good ideas.

There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures.

Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on it. This is often pointless, because usually just checking the output of the last pipe will tell you whether you got what you wanted.

When your script does fail unexpectedly, you'll want to re-run it with at least tracing enabled, so the third line should be something like

  [ "${DEBUG:-0}" = "1" ] && set -x

Re: An almost perfect rsync over SSH backup script

#59

Post by switch007 » Fri, Feb 25, 2022, 6:17 PM UTC

switch007 wrote: post #27

koolba wrote: post #14

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

Not using “env” isn’t “wrong”. It depends how many platforms they want to supply Bet yes pipefail and nounset should definitely be set.

Support*

Re: An almost perfect rsync over SSH backup script

#60

Post by aborsy » Fri, Feb 25, 2022, 6:39 PM UTC

Restic is currently the best solution in my view.

The backup script is much simpler, the repository is properly encrypted, takes snapshots, dedup, mounts the remote, integrates with backends, has clean output, and various useful features for working with repositories.

Rsync over SSH is not even encrypted at rest.