An almost perfect rsync over SSH backup script • Hacker News Board

Re: An almost perfect rsync over SSH backup script

#121

Post by mekster » Sun, Feb 27, 2022, 12:52 AM UTC

aborsy wrote: post #61

aedocw wrote: post #53

Earlier quoted context omitted.

Fully agree, Borg is awesome. I wrote up how I'm using it here[1]. In short, borg backup to a local machine, and that machine uses rclone to copy the backups to an S3 bucket off-site. I've had multiple occasions to restore stuff successfully, and I never have to think about whether my data is retrievable. [1] https://opensource.com/article/17/10/backing-your-machines-b...

You need to run “Borg check” once in a while to check and fix backup integrity errors. S3 egress fees will kill you!

There are several targets for borg that don't charge for traffic. Get off of AWS...

Re: An almost perfect rsync over SSH backup script

#122

Post by mekster » Sun, Feb 27, 2022, 12:58 AM UTC

camnora wrote: post #65

Does anyone have experience with Duplicati [1] or recommend it? I am tentatively looking to make the switch to something a little more sophisticated than manually tar-balling + rsyncing backups. [1] https://www.duplicati.com/

Do yourself a favor and use borg.

https://news.ycombinator.com/item?id=29210222

If you have extra space, you could rclone the result of locally run borg to a cloud storage or find a service that accepts borg directly.

Re: An almost perfect rsync over SSH backup script

#123

Post by pnutjam » Sun, Feb 27, 2022, 3:21 PM UTC

sendfoods wrote: post #23

throwaway81523 wrote: post #3

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

The cheapest I can find on https://www.hetzner.com/storage/storage-box is 3.49 Euro. I assume your talking about the bigger plans with price / TB?

I've been using Time4vps. It's a little bit more expensive, but it's also a full linux box you can do anything with...

I dump my backups and also run a calibre server.

Re: An almost perfect rsync over SSH backup script

#124

Post by linsomniac » Sun, Feb 27, 2022, 4:53 PM UTC

mekster wrote: post #116

linsomniac wrote: post #46

I've been doing rsync-based backups of close to a thousand systems for ~20 years, most notably for a long time I backed up the python.org infrastructure, and I have quite a few thoughts on this. I also have a battle-tested rsync wrapper that I'll point to below. - Backups should be automatic, only requiring attention when it is needed. This script philosophy seems to be "Just do your best, mail a log file, and rely o…

> - In the case that there are no failures, there is no e-mail sent, meaning the user only gets actionable e-mails. I've always thought this isn't the right approach. How do you know if the email server is borked or you commented out the script in cron to debug it and forgot to put it back in? Either there's a weekly status report to tell things have been green or you could place cron checks like healthchecks.io (You…

I get your point, and different people have different things they are comfortable with. Some of that also depends on your environment, in my case I've had this script running nightly backups across hundreds of machines over a dozen years. Sending hundreds of e-mails a day into my inbox isn't going to be workable. But if you have one or two machines, maybe it is. I still don't think so, but again different people.

Things I have done to ensure reliability (again, this core script has been running for a dozen or more years):

- Nagios monitoring of backups: An active check from a monitoring server that alerts if no recent successful backups.

- "paper path" monitoring of e-mail: Send an e-mail to an external mailbox and have an active check in Nagios that reports if it has not seen an e-mail recently.

- With hundreds of machines, we were in the management interface enough (not daily, but at least monthly) that we would tend to notice before TOO long if something was out of whack.

- Regular backup audits: We would perform quarterly backup audits of the important machines, we had a whole workflow for those, which would also give us confidence that the backups were running as expected and that if something got out of whack it didn't go too long. Many of these depend on your definition of "too long".

As far as "zfs send", I totally agree. However, even today I have very few machines other than my backup machines that are running ZFS, so that's not really an option for these backups.

Re: An almost perfect rsync over SSH backup script

#125

Post by 0xbadcafebee » Wed, Mar 02, 2022, 3:15 PM UTC

mekster wrote: post #117

0xbadcafebee wrote: post #58

Earlier quoted context omitted.

#!/usr/bin/env bash and set -u are always good ideas. There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures. Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on i…

> There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures. You don't let go of -e for that. dont_mind_failure || true important_process add 'true' specifically if you must.

There are scripts where you don't want any failure to kill the script. You would have to wrap nearly every line in that kind of exception handling, and if you miss one, the script fails. It is much simpler to omit -e in that case

Re: An almost perfect rsync over SSH backup script

#126

Post by uhradone » Mon, Mar 07, 2022, 11:08 AM UTC

throwaway81523 wrote: post #3

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

As a small film production we have approx. 120 TB to backup. Thats a lot of money per month, if you check AWS it is a fortune (half the value of a car) per month.

Re: An almost perfect rsync over SSH backup script

#127

Post by uhradone » Mon, Mar 07, 2022, 11:27 AM UTC

gunapologist99 wrote: post #96

The script could use --rsync-path='sudo rsync', so that your rsync backup script can access root-owned files on the remote system . It looks like you can add that to the RSYNCCONF part of "subpart of part 2".

For security reasons the user "backup" on the linux acting backup system is not part of the sudo users. Having sudo rights will to my knowledge not help to excess root files on the remote system?

Re: An almost perfect rsync over SSH backup script

#128

Post by uhradone » Mon, Mar 07, 2022, 11:33 AM UTC

rurban wrote: post #2

Perfect? Ha. Wrong ps grep usage. MacOS specific. No shell quoting. At least he doesn't advocate rm -rf / by an incorrect usage of --delete.

You missed the word "almost" just before the word "perfect": meaning there is space for improvementes and any coding has, if you dig deeper, a never ending context. So, you see something "not so or not at all perfect" just name it to improve it. THX

Re: An almost perfect rsync over SSH backup script

#129

Post by uhradone » Mon, Mar 07, 2022, 12:16 PM UTC

darrenf wrote: post #19

Can't help but throw a couple cents/pennies around whenever an "almost perfect" or "advanced" script makes my knee jerk so wildly. So here goes. For this part: pro_on=$(ps aux | grep -c rsync) # if someone is using rsync # grep is also producing one entry so -gt 1 `grep` can be excluded from the output by putting square brackets around one character of the search pattern, like so: pro_on=$(ps aux | grep -c rsyn[c]) A…

THX the "almost" is getting closer to "almost"!

Re: An almost perfect rsync over SSH backup script

#130

Post by uhradone » Mon, Mar 07, 2022, 1:12 PM UTC

branon wrote: post #21

Haven't read the page yet but the header image struck me as interesting. "Molex to SATA, lose all your data" was drilled into my head at an early job and appears to be what's depicted there. If your disks are set up in a similar way, you might have a very immediate need for a good backup script :)

Any SATA connector that is powering the drive has little pins and can be not well made and cause fire. Yes, on the Molex site is a big amount of amperes that can go through. But you hardly find a PSU with 24 direct outputs for HDDs.