Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An almost perfect rsync over SSH backup script

blog.zazu.berlin

Post reply on HN
Print view View original

101–110 of 130 posts

Re: An almost perfect rsync over SSH backup script

#101

Post by Jenda_ »

chronomex wrote: post #72
adrian_b wrote: post #30

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most f…

Rsync also doesn't track hardlinks by default! I use "-avPzHAXS" for my backups.

Also have a look at --numeric-ids, otherwise it translates UIDs back and forth according to real user names - which will end up in a terrible mess, especially when you are restoring the backup from a live "CD" (which has different UID ←→ username mapping than the target system).

Re: An almost perfect rsync over SSH backup script

#102

Post by [deleted] »

chronomex wrote: post #72
adrian_b wrote: post #30

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most f…

Rsync also doesn't track hardlinks by default! I use "-avPzHAXS" for my backups.

[deleted]

Re: An almost perfect rsync over SSH backup script

#103

Post by amelius »

Jenda_ wrote: post #101
chronomex wrote: post #72

Earlier quoted context omitted.

Rsync also doesn't track hardlinks by default! I use "-avPzHAXS" for my backups.

Also have a look at --numeric-ids, otherwise it translates UIDs back and forth according to real user names - which will end up in a terrible mess, especially when you are restoring the backup from a live "CD" (which has different UID ←→ username mapping than the target system).

I use getfacl to save permissions correctly, in case the backup server uses different user/group mappings. And setfacl after restoring.

(I wish this was handled by rsync.)

Re: An almost perfect rsync over SSH backup script

#104

Post by varenc »

Symbiote wrote: post #28
joosters wrote: post #25

Earlier quoted context omitted.

> `grep` can be excluded from the output by putting square brackets around one character of the search pattern That's a great trick, thanks! I'm so used to adding '| grep -v grep' at the end of a 'ps | grep' command, your way is much nicer.

On Linux, pgrep is probably what you need most of the time, or pkill.

I like the `psgrep` tool which behaves much more like greping ps that pgrep does. And `psgrep` is actually just a bash script itself.

Re: An almost perfect rsync over SSH backup script

#105

Post by sneak »

beepbooptheory wrote: post #83
justin_oaks wrote: post #56

I use Restic ( https://restic.net/ ) for backups. Encrypted backup, incremental, deduplicating, and supports many storage backends. Most recent discussion on Hacker News: https://news.ycombinator.com/item?id=29209455

I like restic, but doing rsync scripts like this always win out for me because I find myself wanting to stage the backups on the destination systems, and not the source/prod ones.

I really, really wanted to love restic, but it makes far, far too many tiny size files for my main remote backup use-case: backing up dozens of TB offsite to Glacier Deep Archive.

Eagerly awaiting a configurable chunk size, if they decide to do that.

Re: An almost perfect rsync over SSH backup script

#107

Post by deckard1 »

LeoPanthera wrote: post #92
LoveGracePeace wrote: post #89

Earlier quoted context omitted.

I've been doing my scripts for decades on various, numerous, Linux and UNIX systems, real and virtual machines and never used env shell nor have I seen "set -euo pipefail" and have had zero issues. Saying "The first two lines of the script are already wrong;" is wrong. Is that better? IDK, maybe. But "#!/bin/bash" works fine.

> "#!/bin/bash" works fine It doesn't work at all on any BSD OS, which does not store bash in /bin - instead it is in /usr/local/bin Specifying "env bash" makes it work on any UNIX, since the location of env is a constant, unlike bash.

does it matter? MacOS is using some ancient ass 3.x bash from 2007. The location of bash is the least of your worries when it comes to portability. There's no guarantee any of the commands in your script (a) exist on the system or (b) work with the same options and flags that the script uses. I don't even know how many "netcat" commands I've seen in the wild. You could be running in some BusyBox or pared down Docker container. Blindly running a script that hasn't been deliberately crafted to work on your system is just asking for trouble.

Re: An almost perfect rsync over SSH backup script

#108

Post by shaicoleman »

sneak wrote: post #105
beepbooptheory wrote: post #83

Earlier quoted context omitted.

I like restic, but doing rsync scripts like this always win out for me because I find myself wanting to stage the backups on the destination systems, and not the source/prod ones.

I really, really wanted to love restic, but it makes far, far too many tiny size files for my main remote backup use-case: backing up dozens of TB offsite to Glacier Deep Archive. Eagerly awaiting a configurable chunk size, if they decide to do that.

Perhaps Kopia will work for your use case:

https://kopia.io/

https://kopia.io/docs/advanced/amazon-s3/

Re: An almost perfect rsync over SSH backup script

#109

Post by ranger_danger »

adrian_b wrote: post #30

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most f…

Probably because 99% of users do not have or need xattrs/acls.

Re: An almost perfect rsync over SSH backup script

#110

Post by longcommonname »

eurasiantiger wrote: post #93
koolba wrote: post #33

Earlier quoted context omitted.

Using -e isn't an excuse to not understand how it works and when it doesn't. It's so that the default in most situation is to exit on failure as that's likely what you want to do. That leads to more terse scripts which hopefully are easier to write correctly and understand. I'd argue that opting in to exit-on-failure for stand alone commands is the right default.

No, absolutely wrong. The ”right default” is to catch return codes and provide actionable, clear and consistent error messages through an error-exit function.

If I'm writing a bash script I run manually and it's more than a handful of functions I agree. I'm there to babysit it and its probably complicated.

If the bash script is ran on thousands of containers where its not possible to babysit. My number one job is to stop immediately when an error happens and surface that error to any monitoring system.

Post reply on HN