Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An almost perfect rsync over SSH backup script

blog.zazu.berlin

Post reply on HN
Print view View original

111–120 of 130 posts

Re: An almost perfect rsync over SSH backup script

#111

Post by remram »

shaicoleman wrote: post #108
sneak wrote: post #105

Earlier quoted context omitted.

I really, really wanted to love restic, but it makes far, far too many tiny size files for my main remote backup use-case: backing up dozens of TB offsite to Glacier Deep Archive. Eagerly awaiting a configurable chunk size, if they decide to do that.

Perhaps Kopia will work for your use case: https://kopia.io/ https://kopia.io/docs/advanced/amazon-s3/

Kopia looks great! I am currently using Duplicati for my personal machines, but Mono and its dependencies have been less than reliable. borg/restic/... look good but seem like only part of a solution, having to write my own scripts and crontabs with monitoring etc on top seems counter-productive.

edit: No web interface?

Re: An almost perfect rsync over SSH backup script

#112

Post by corobo »

klysm wrote: post #13
corobo wrote: post #11

Earlier quoted context omitted.

Make sure your borg repos are copying properly and in full! I had a horrible realisation that my borg backups were timing out on the offiste copy, meaning the resulting offsite backup I had was non-existent. The heartbreaking error message Inconsistency detected. Please run "borg check [repository]" - although likely this is "beyond repair" Course following the 3-2-1 rule you're probably good, but aye I'm treating bo…

This is why you gotta test your backups too

Indeed. You only get burned once

.. and then apparently once again a decade later when you let your guard down

Re: An almost perfect rsync over SSH backup script

#113

Post by corobo »

dspillett wrote: post #26
corobo wrote: post #11

Earlier quoted context omitted.

Make sure your borg repos are copying properly and in full! I had a horrible realisation that my borg backups were timing out on the offiste copy, meaning the resulting offsite backup I had was non-existent. The heartbreaking error message Inconsistency detected. Please run "borg check [repository]" - although likely this is "beyond repair" Course following the 3-2-1 rule you're probably good, but aye I'm treating bo…

> I had a horrible realisation that my borg backups were timing out ... meaning the ... backup I had was ... A backup that isn't tested is not a true backup, it is a disappointment waiting to be found! This can happy with any backup tool, my hand-crafted¹ rsync based scripts included. Testing isn't hard to setup if you don't mind the final step being manual. Snapshots have a checksum file, and daily one is picked and…

> A backup that isn't tested is not a true backup, it is a disappointment waiting to be found

Yup. 500 gigs of disk seatfiller (a full backup in this instance would have been somewhere between 1-2T)

Live and learn.. and learn again later when you let yourself slip :)

My backup process is a LOT noisier if any of the commands fail/timeout/dirs-arent-same-size-after now!

Re: An almost perfect rsync over SSH backup script

#114

Post by eurasiantiger »

longcommonname wrote: post #110
eurasiantiger wrote: post #93

Earlier quoted context omitted.

No, absolutely wrong. The ”right default” is to catch return codes and provide actionable, clear and consistent error messages through an error-exit function.

If I'm writing a bash script I run manually and it's more than a handful of functions I agree. I'm there to babysit it and its probably complicated. If the bash script is ran on thousands of containers where its not possible to babysit. My number one job is to stop immediately when an error happens and surface that error to any monitoring system.

I agree about that distinction, but still, won’t that error need to be formatted? Is it safe to rely on logging picking up on the error or does the ”simple script” solution imply that there is monitoring for the script exit code?

Re: An almost perfect rsync over SSH backup script

#115

Post by dspillett »

corobo wrote: post #113
dspillett wrote: post #26

Earlier quoted context omitted.

> I had a horrible realisation that my borg backups were timing out ... meaning the ... backup I had was ... A backup that isn't tested is not a true backup, it is a disappointment waiting to be found! This can happy with any backup tool, my hand-crafted¹ rsync based scripts included. Testing isn't hard to setup if you don't mind the final step being manual. Snapshots have a checksum file, and daily one is picked and…

> A backup that isn't tested is not a true backup, it is a disappointment waiting to be found Yup. 500 gigs of disk seatfiller (a full backup in this instance would have been somewhere between 1-2T) Live and learn.. and learn again later when you let yourself slip :) My backup process is a LOT noisier if any of the commands fail/timeout/dirs-arent-same-size-after now!

> Live and learn.. and learn again later when you let yourself slip :)

Definitely. I'm as careful as I am due to past issues, either my own or those of others that I've witnessed. Seeing the look on someone's face when they ask “You know about these things, you can do something to get it back right? Right?!”, and having to let them down…

Re: An almost perfect rsync over SSH backup script

#116

Post by mekster »

linsomniac wrote: post #46

I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I have quite a few thoughts on this. I also have a battle-tested rsync wrapper that I'll point to below. - Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely o…

> - In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails.

I've always thought this isn't the right approach.

How do you know if the email server is borked or you commented out the script in cron to debug it and forgot to put it back in?

Either there's a weekly status report to tell things have been green or you could place cron checks like healthchecks.io (You can self host it.)

Also it's much better to use 'zfs send' with large volumes to backup if both ends have zfs as zfs knows which files have changed and it doesn't have to scan for what has changed on each go as any other tools do like rsync.

https://arstechnica.com/information-technology/2015/12/rsync...

Re: An almost perfect rsync over SSH backup script

#117

Post by mekster »

0xbadcafebee wrote: post #58
koolba wrote: post #14

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

#!/usr/bin/env bash and set -u are always good ideas. There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures. Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on i…

> There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures.

You don't let go of -e for that.

  dont_mind_failure || true
  important_process
add 'true' specifically if you must.

Re: An almost perfect rsync over SSH backup script

#118

Post by mekster »

deckard1 wrote: post #107
LeoPanthera wrote: post #92

Earlier quoted context omitted.

> "#!/bin/bash" works fine It doesn't work at all on any BSD OS, which does not store bash in /bin - instead it is in /usr/local/bin Specifying "env bash" makes it work on any UNIX, since the location of env is a constant, unlike bash.

does it matter? MacOS is using some ancient ass 3.x bash from 2007. The location of bash is the least of your worries when it comes to portability. There's no guarantee any of the commands in your script (a) exist on the system or (b) work with the same options and flags that the script uses. I don't even know how many "netcat" commands I've seen in the wild. You could be running in some BusyBox or pared down Docker…

You call 2007 ancient when bash has been around for over 30 years?

What exactly has changed in the last 15 years?

Re: An almost perfect rsync over SSH backup script

#120

Post by mekster »

justin_oaks wrote: post #56

I use Restic ( https://restic.net/ ) for backups. Encrypted backup, incremental, deduplicating, and supports many storage backends. Most recent discussion on Hacker News: https://news.ycombinator.com/item?id=29209455

Sorry to say none of these modern backup tools work reliably when backup volumes get large except borg.

https://news.ycombinator.com/item?id=29210222

Using zfs snapshot is the best way to go but having a cloud destination wouldn't be easy for these tools unfortunately but there are services that accept borg and zfs send.

Post reply on HN