Live data from Hacker News

An almost perfect rsync over SSH backup script

blog.zazu.berlin

31–40 of 130 posts

Re: An almost perfect rsync over SSH backup script

#31
post #21

Haven't read the page yet but the header image struck me as interesting. "Molex to SATA, lose all your data" was drilled into my head at an early job and appears to be what's depicted there. If your disks are set up in a similar way, you might have a very immediate need for a good backup script :)

That's a weird mantra, molex to sata is perfectly fine.

Re: An almost perfect rsync over SSH backup script

#32

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

I used to do something similarly complex to the OP with ssh and --link-dest to make it share inodes so I could easily keep N days of backups with file level deduplication [1].

Then I moved to Borg and haven't looked back. It does the same end goal in a better way, is way faster, and easier to work with overall [2].

[1]: https://nuxx.net/blog/2009/12/06/time-machine-for-freebsd/ [2]: https://nuxx.net/blog/2019/11/10/using-borg-for-backing-up-n...

Re: An almost perfect rsync over SSH backup script

#33
post #29
post #14

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

I wish this misguided notion regarding "inofficial strict mode" would go away, as `set -e` is suboptimal (and can be very tedious to deal with on top of that): https://mywiki.wooledge.org/BashFAQ/105 It makes sense to opt into errexit for select blocks/sections in scripts and under certain circumstances, but having it default-on is a recipe for quite a bit of head-scratching in the future.

Using -e isn't an excuse to not understand how it works and when it doesn't. It's so that the default in most situation is to exit on failure as that's likely what you want to do. That leads to more terse scripts which hopefully are easier to write correctly and understand. I'd argue that opting in to exit-on-failure for stand alone commands is the right default.

Re: An almost perfect rsync over SSH backup script

#34
For rsync tasks that require scheduling and need to ensure only a single instance executes, I personally prefer creating a systemd service (type=oneshot) and have that run under a systemd timer (set OnBootSec, OnUnitActiveSec)

https://wiki.archlinux.org/title/Systemd/Timers

EDIT: I'd love to hear if anyone knows of any downsides, edge cases for running rsync under systemd timers.

Re: An almost perfect rsync over SSH backup script

#35

Too complicated to even try to read. Rsync is great but I've switched to Borg for backups. Borg isn't perfect but it is a higher level approach to backups, as it were. Hetzner recently dropped the price of their Storage Box backup product to about 2 euro per TB per month, and Borg works nicely with it. Borg encrypts all the backup contents and conceals the metadata on the backup server, and yet you can (with the encr…

I used rsync 20 years ago, switched to Borg, then switched to Restic. It’s nice to send backups to destinations like B2 or S3 without any special requirements. But the Hetzner Storage Box is good deal.

Now all I need is better client side software for these things, something I could use on my moms computer. Vorta is not that good, although I commend the effort.

Until the UI side is sorted, my moms laptop stays on Backblaze.

Re: An almost perfect rsync over SSH backup script

#36
post #9
post #6

rsnapshot[1] is what I used on FreeBSD with a snapshot. It is like a well-tested version of the author's rsync and ssh blog post. I have a blog post here[2] describing my setup. It saved my bacon multiple times. Also, rsnapshot works in pull mode, so no client is needed on my Linux/macOS desktop or server except ssh and rsync. [1] https://github.com/rsnapshot/rsnapshot [2] https://www.cyberciti.biz/faq/howto-install-…

Yep. We use rsnapshot with btrfs snapshots on Debian / Ubuntu. You can set up multiple generations easily. Efficient storage using hard links. Very reliable.

AFAIK the crucial part of this, is rsync's `--link-dest=DIR` parameter, which hard-links an unmodified file to the respective place in "DIR", rather than making a copy.

Re: An almost perfect rsync over SSH backup script

#37
post #21

Haven't read the page yet but the header image struck me as interesting. "Molex to SATA, lose all your data" was drilled into my head at an early job and appears to be what's depicted there. If your disks are set up in a similar way, you might have a very immediate need for a good backup script :)

could you explain why it's a bad idea to use molex to sata?

did your guys use shitty injection molded connectors that melted or is there another reason?

Re: An almost perfect rsync over SSH backup script

#40
post #9
post #6

rsnapshot[1] is what I used on FreeBSD with a snapshot. It is like a well-tested version of the author's rsync and ssh blog post. I have a blog post here[2] describing my setup. It saved my bacon multiple times. Also, rsnapshot works in pull mode, so no client is needed on my Linux/macOS desktop or server except ssh and rsync. [1] https://github.com/rsnapshot/rsnapshot [2] https://www.cyberciti.biz/faq/howto-install-…

Yep. We use rsnapshot with btrfs snapshots on Debian / Ubuntu. You can set up multiple generations easily. Efficient storage using hard links. Very reliable.

Could you elaborate on where the btrfs snapshots come into play? I use rsnapshot on a plain ext4 fs and it’s pretty slow (lots of small files).
Post reply on HN