Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An almost perfect rsync over SSH backup script

blog.zazu.berlin

Post reply on HN
Print view View original

91–100 of 130 posts

Re: An almost perfect rsync over SSH backup script

#91

Post by adrian_b »

throwawayboise wrote: post #77
adrian_b wrote: post #30

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most f…

This also presumes that the target's rsync implementation and filesystem supports the same metadata.

Most modern file systems have been updated to support equivalent metadata.

However many old versions of file systems lacked support for things like high-resolution timestamps, extended attributes or access-control lists. Because such support has been introduced much later, there are a lot of programs for archiving, backup and copying which lose such metadata, at least with their default options.

While equivalent metadata exists on Linux XFS and EXT4, Windows NTFS, FreeBSD UFS and ZFS etc. each platform has different APIs for accessing file metadata (and on Linux the API may vary even between the many available file systems), so only few programs, like rsync, are able to copy everything while taking care to use the correct API to replicate the metadata without losing information.

Re: An almost perfect rsync over SSH backup script

#92

Post by LeoPanthera »

LoveGracePeace wrote: post #89
koolba wrote: post #14

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

I've been doing my scripts for decades on various, numerous, Linux and UNIX systems, real and virtual machines and never used env shell nor have I seen "set -euo pipefail" and have had zero issues. Saying "The first two lines of the script are already wrong;" is wrong. Is that better? IDK, maybe. But "#!/bin/bash" works fine.

> "#!/bin/bash" works fine

It doesn't work at all on any BSD OS, which does not store bash in /bin - instead it is in /usr/local/bin

Specifying "env bash" makes it work on any UNIX, since the location of env is a constant, unlike bash.

Re: An almost perfect rsync over SSH backup script

#93

Post by eurasiantiger »

koolba wrote: post #33
c0l0 wrote: post #29

Earlier quoted context omitted.

I wish this misguided notion regarding "inofficial strict mode" would go away, as `set -e` is suboptimal (and can be very tedious to deal with on top of that): https://mywiki.wooledge.org/BashFAQ/105 It makes sense to opt into errexit for select blocks/sections in scripts and under certain circumstances, but having it default-on is a recipe for quite a bit of head-scratching in the future.

Using -e isn't an excuse to not understand how it works and when it doesn't. It's so that the default in most situation is to exit on failure as that's likely what you want to do. That leads to more terse scripts which hopefully are easier to write correctly and understand. I'd argue that opting in to exit-on-failure for stand alone commands is the right default.

No, absolutely wrong. The ”right default” is to catch return codes and provide actionable, clear and consistent error messages through an error-exit function.

Re: An almost perfect rsync over SSH backup script

#94

Post by gunapologist99 »

It's nice that the script works both ways, but keep in mind that it's always best to pull backups from a remote system, so that an attacker in the production system can't log into the backup system and delete the backups.

If your prod system(s) is/are logging into the backup system, then you have a big problem because if any of them are compromised, they can wipe out / corrupt / etc all of their own backups, and possibly the other server's backups as well (if the same user account is used for all of them.) This problem goes away if your (isolated, hardened) backup server logs into the prod systems. Of course, the inverse problem is that if an attacker manages to get into your backup system, then they also have (at least read) access to all of your prod systems!

Re: An almost perfect rsync over SSH backup script

#95

Post by Quasimarion »

colejohnson66 wrote: post #85
chalst wrote: post #81

Earlier quoted context omitted.

Better than rsync is using support for snapshots and replication in the filesystem, such as described for ZFS in https://serverfault.com/questions/842531/how-to-perform-incr... Btrfs and apfs also support auto-snapshotting and replication, but I don't have experience with them.

ZFS snapshots also have the added benefit of not taking up duplicate space. And, should a Linux ransomware ever come into existence, snapshots help against them.

Btrfs and apfs also support copy-on-write snapshots.

Also, snapshots are NOT backups. And one definitely should not rely on snapshots along to keep their files safe.

Re: An almost perfect rsync over SSH backup script

#98

Post by quesera »

Sohcahtoa82 wrote: post #73

About 10 years ago, I was earning my CS degree, and my Intro to Unix teacher was showing us how to use rsync. He somehow made an error that synced an empty directory to his home directory, deleting everything from his home directory. The lesson I learned in that class was to not use rsync, sadly.

> The lesson I learned in that class was to not use rsync, sadly.

This is a poor lesson! If your TA tripped on the stairs and broke their nose, would you swear off stairs forever?

rsync is an essential tool for anyone who works with files in the UNIX world. There are other options, sometimes, but rsync is almost-always present, and does almost-everything you usually want.

Any tool can be used incorrectly, but rsync is easy and follows the same pattern as most other UNIX file copy tools. If you can use cp, you can use rsync safely.

Re: An almost perfect rsync over SSH backup script

#99

Post by colejohnson66 »

Quasimarion wrote: post #95
colejohnson66 wrote: post #85

Earlier quoted context omitted.

ZFS snapshots also have the added benefit of not taking up duplicate space. And, should a Linux ransomware ever come into existence, snapshots help against them.

Btrfs and apfs also support copy-on-write snapshots. Also, snapshots are NOT backups. And one definitely should not rely on snapshots along to keep their files safe.

They aren’t; a backup should preferably be offsite, but nothing is perfect

Re: An almost perfect rsync over SSH backup script

#100

Post by uhradone »

adrian_b wrote: post #30

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most f…

Hey Adrian, thanks for the improvements and hints!
Post reply on HN