Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An almost perfect rsync over SSH backup script

blog.zazu.berlin

Post reply on HN
Print view View original

81–90 of 130 posts

Re: An almost perfect rsync over SSH backup script

#81

Post by chalst »

adrian_b wrote: post #30

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most f…

Better than rsync is using support for snapshots and replication in the filesystem, such as described for ZFS in

https://serverfault.com/questions/842531/how-to-perform-incr...

Btrfs and apfs also support auto-snapshotting and replication, but I don't have experience with them.

Re: An almost perfect rsync over SSH backup script

#82

Post by bobbyi »

senko wrote: post #42
koolba wrote: post #14

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

Agreed on fail on error, but the first one is needlessly pedantic. Find me a single Linux distro where bash, if installed, is not available in /bin

On my laptop (admittedly, it's running Mac OS, not a linux distro), /bin/bash is version 3.2.57 released in 2007, while '/usr/bin/env bash' invokes bash 5.1.8, released in 2020.

Re: An almost perfect rsync over SSH backup script

#83

Post by beepbooptheory »

justin_oaks wrote: post #56

I use Restic ( https://restic.net/ ) for backups. Encrypted backup, incremental, deduplicating, and supports many storage backends. Most recent discussion on Hacker News: https://news.ycombinator.com/item?id=29209455

I like restic, but doing rsync scripts like this always win out for me because I find myself wanting to stage the backups on the destination systems, and not the source/prod ones.

Re: An almost perfect rsync over SSH backup script

#84

Post by tombrossman »

adrian_b wrote: post #30

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most f…

For years I had a custom script sync my ~/.ssh directory on my primary workstation to my laptop, to pick up new keys and config changes. It failed after I switched from Ubuntu to Fedora, and I was surprised to discover --xattrs fixed it.

tl;dr try this if rsync fails in unexpected ways:

  echo "Syncing ~/.ssh directory"
  rsync --archive --delete --xattrs ~/.ssh/ laptop:.ssh/

Re: An almost perfect rsync over SSH backup script

#85

Post by colejohnson66 »

chalst wrote: post #81
adrian_b wrote: post #30

Beware that this script only uses rsync with the "--archive" flag. This may be enough for some users, but "--archive" does not copy all file metadata, so it may cause surprises. rsync must be invoked with "--archive --xattrs --acls" to guarantee complete file copies. Unfortunately all command-line utilities for file copying that are available for UNIX-like operating systems use default options that do not copy most f…

Better than rsync is using support for snapshots and replication in the filesystem, such as described for ZFS in https://serverfault.com/questions/842531/how-to-perform-incr... Btrfs and apfs also support auto-snapshotting and replication, but I don't have experience with them.

ZFS snapshots also have the added benefit of not taking up duplicate space. And, should a Linux ransomware ever come into existence, snapshots help against them.

Re: An almost perfect rsync over SSH backup script

#86

Post by scottlamb »

jethro_tell wrote: post #67
senko wrote: post #42

Earlier quoted context omitted.

Agreed on fail on error, but the first one is needlessly pedantic. Find me a single Linux distro where bash, if installed, is not available in /bin

As a system eng for almost 15 years I've spent a good number of years handling deployments, dependancies, environment evaluations and all the little minutiae that's required to run large complex distributed systems at scale. This is fine to ignore if it's your personal box and no one else is working on it. However I've seen it happen quite frequently in systems that were designed with container like 'chroot-lite' pro…

> This is why we have industry best practices, so that people who don't understand why something exists can just follow the best practices and we don't all have to be experts in things outside of our direct field.

We don't have industry best practices for shell scripts, no matter what consultants / HN commenters with strong opinions say. You can see in this thread that folks disagree about what the best practices are. It's worth paying attention to folks' rationale for their opinions—I learned some caveats about "-e" from following the links here. But the talk about "table stakes" and "industry best practices" is (extended bleep). Those don't exist.

Different projects/companies may have their own best practice guides that make sense in their environment. senko mentioned Google data centers as a place where it might make sense to be more rigorous. Google's guide says to use "#!/bin/bash". [1] If that doesn't work in your environment, fine, but that doesn't make them wrong.

Shell is a surprisingly and unnecessarily difficult language to write correctly. To the extent there is a best practice on it, I think it's "use a better language for anything that might become large or important". The Google style guide I linked says more or less the same thing near the beginning. The subtleties discussed in the rest give you a taste of why...

[1] https://google.github.io/styleguide/shellguide.html

Re: An almost perfect rsync over SSH backup script

#87

Post by branon »

kobalsky wrote: post #37
branon wrote: post #21

Haven't read the page yet but the header image struck me as interesting. "Molex to SATA, lose all your data" was drilled into my head at an early job and appears to be what's depicted there. If your disks are set up in a similar way, you might have a very immediate need for a good backup script :)

could you explain why it's a bad idea to use molex to sata? did your guys use shitty injection molded connectors that melted or is there another reason?

I didn't really know why until I read scottlamb's comment, and yours.

I think this is the only plausible explanation - someone had a bad experience with a really crappy connector and it stuck.

Re: An almost perfect rsync over SSH backup script

#88

Post by sitkack »

sgt wrote: post #80
sitkack wrote: post #76

Earlier quoted context omitted.

This SO post goes over the semantics of `--inplace` and `--no-whole-file` --inplace update destination files in-place --whole-file, -W copy files whole (w/o delta-xfer algorithm) It is very important to debug rsync scripts with a test corpus and not on live data, preferably on test vm or container. https://superuser.com/questions/576035/does-rsync-inplace-wr... Also, when transferring from different systems, make sur…

HN's new pet word; footgun

https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=tru...

Re: An almost perfect rsync over SSH backup script

#89

Post by LoveGracePeace »

koolba wrote: post #14

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

I've been doing my scripts for decades on various, numerous, Linux and UNIX systems, real and virtual machines and never used env shell nor have I seen "set -euo pipefail" and have had zero issues.

Saying "The first two lines of the script are already wrong;" is wrong. Is that better? IDK, maybe. But "#!/bin/bash" works fine.

Re: An almost perfect rsync over SSH backup script

#90

Post by lamontcg »

BeefWellington wrote: post #16
rurban wrote: post #2

Perfect? Ha. Wrong ps grep usage. MacOS specific. No shell quoting. At least he doesn't advocate rm -rf / by an incorrect usage of --delete.

> Perfect? Ha. The cynic in me believes this was intentional to drive engagement and discussion. It's that old adage of "if you want to learn something just say something wrong about it on the Internet" at work.

There's a million blog posts out there with "the perfect setup" which is this circle jerking way of saying that its the author's preferred configuration.

I don't entirely know what the motivation is, but as a grumpy old GenXer, I wish it would stop.

Post reply on HN