Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An almost perfect rsync over SSH backup script

blog.zazu.berlin

Post reply on HN
Print view View original

71–80 of 130 posts

Re: An almost perfect rsync over SSH backup script

#71

Post by throwaway81523 »

lvass wrote: post #10
throwaway81523 wrote: post #3

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

>Storage Box backup product The script here really only makes sense for servers you physically own. You wouldn't accept SSH access from a key located on a plain VPS, right? Also this script doesn't seem to encrypt the data at all!! Very dangerous on a VPS.

> You wouldn't accept SSH access from a key located on a plain VPS, right?

With Borg, I use ssh -A from my laptop to start backups going. I can think of some other schemes like using multiple user accounts on a single Storage Box (Hetzner supports that and I believe offers an API) so that different VPS can't clobber each other's backups, that old backups become read-only, etc. It might be interesting to add some finer access controls on the server side. Borg supports an append-only mode but right now, that's only a config option rather than a security setting, I believe.

I've only recently started using Borg so I'm not really familiar with its intricacies yet. There are some things I would change but it is mostly well thought out, imho.

Re: An almost perfect rsync over SSH backup script

#72

Post by chronomex »

adrian_b wrote: post #30

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most f…

Rsync also doesn't track hardlinks by default! I use "-avPzHAXS" for my backups.

Re: An almost perfect rsync over SSH backup script

#73

Post by Sohcahtoa82 »

About 10 years ago, I was earning my CS degree, and my Intro to Unix teacher was showing us how to use rsync. He somehow made an error that synced an empty directory to his home directory, deleting everything from his home directory.

The lesson I learned in that class was to not use rsync, sadly.

Re: An almost perfect rsync over SSH backup script

#74

Post by throwaway81523 »

rsync wrote: post #51
throwaway81523 wrote: post #3

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

"Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it." I'm still not clear - does the Hetzner storage box have the borg binary installed on their end ? As in, one could run: ssh user@hetzner borg --version ... or are you accessing borg over an sshfs mount, etc. ? Asking for a friend ...

You can't run arbitrary shell commands on the storage box server. They have the Borg binary (1.17 last time I looked) installed on the server, and they officially support it. They seem to specially recognize the borg commands.

Re: An almost perfect rsync over SSH backup script

#75

Post by aidenn0 »

senko wrote: post #42
koolba wrote: post #14

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

Agreed on fail on error, but the first one is needlessly pedantic. Find me a single Linux distro where bash, if installed, is not available in /bin

1. bash runs on more than just linux

2. Many non-LSB distros

Re: An almost perfect rsync over SSH backup script

#76

Post by sitkack »

linsomniac wrote: post #46

I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I have quite a few thoughts on this. I also have a battle-tested rsync wrapper that I'll point to below. - Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely o…

This SO post goes over the semantics of `--inplace` and `--no-whole-file`

  --inplace                update destination files in-place
  --whole-file, -W         copy files whole (w/o delta-xfer algorithm)
It is very important to debug rsync scripts with a test corpus and not on live data, preferably on test vm or container.

https://superuser.com/questions/576035/does-rsync-inplace-wr...

Also, when transferring from different systems, make sure that both rsyncs are of a high enough version, again preferably the same.

MacOS ships with a really old version of rsync that doesn't support extended attributes at `/usr/bin/rsync`

   rsync  version 2.6.9  protocol version 29
This is kinda a large footgun.

Re: An almost perfect rsync over SSH backup script

#77

Post by throwawayboise »

adrian_b wrote: post #30

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most f…

This also presumes that the target's rsync implementation and filesystem supports the same metadata.

Re: An almost perfect rsync over SSH backup script

#78

Post by senko »

aidenn0 wrote: post #75
senko wrote: post #42

Earlier quoted context omitted.

Agreed on fail on error, but the first one is needlessly pedantic. Find me a single Linux distro where bash, if installed, is not available in /bin

1. bash runs on more than just linux 2. Many non-LSB distros

The article talks about Linux.

Re: An almost perfect rsync over SSH backup script

#79

Post by senko »

jethro_tell wrote: post #67
senko wrote: post #42

Earlier quoted context omitted.

Agreed on fail on error, but the first one is needlessly pedantic. Find me a single Linux distro where bash, if installed, is not available in /bin

As a system eng for almost 15 years I've spent a good number of years handling deployments, dependancies, environment evaluations and all the little minutiae that's required to run large complex distributed systems at scale. This is fine to ignore if it's your personal box and no one else is working on it. However I've seen it happen quite frequently in systems that were designed with container like 'chroot-lite' pro…

I was a sysadmin on SunOS, HP/UX and BSD systems back in the day and have been maintaining various Linux-based systems in the past few decades as well.

We're not talking about Google data center here, it's just someone's shell script.

Re: An almost perfect rsync over SSH backup script

#80

Post by sgt »

sitkack wrote: post #76
linsomniac wrote: post #46

I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I have quite a few thoughts on this. I also have a battle-tested rsync wrapper that I'll point to below. - Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely o…

This SO post goes over the semantics of `--inplace` and `--no-whole-file` --inplace update destination files in-place --whole-file, -W copy files whole (w/o delta-xfer algorithm) It is very important to debug rsync scripts with a test corpus and not on live data, preferably on test vm or container. https://superuser.com/questions/576035/does-rsync-inplace-wr... Also, when transferring from different systems, make sur…

HN's new pet word; footgun
Post reply on HN