Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Saf – simple, reliable, rsync-based, battle tested, rounded backup

github.com

Post reply on HN
Print view View original

21–26 of 26 posts

Re: Show HN: Saf – simple, reliable, rsync-based, battle tested, rounded backup

#21

Post by binaryapparatus »

gizmo wrote: post #18

How does it deal with interrupted backups? Can it automatically prune backups older than N days? I don’t see anything about encryption.

> How does it deal with interrupted backups?

Any new backup is hardlinked against previous in temporary 'in-progress' directory, then renamed to proper name at the end. If backup breaks, new 'saf backup' by default first removes 'in-progress' than starts things again (linking with latest good one) but you can 'saf backup --resume' to try to finish interrupted one. I prefer clean try again (which is the default) but --resume works well too.

> Can it automatically prune backups older than N days?

Yes, manually by 'saf prune' on top of 'saf backup' doing prune itself. Prune periods are defined in each .saf.conf, per backup source location, with the defaults of 2/30/60/730/3650 days, for all/daily/weekly/monthly/yearly backups. All defaults are easy to change per source.

> I don’t see anything about encryption.

saf doesn't deal with encryption, only with transport. I prefer to use other specialized tool for the encryption if I have such backup target that needs one.

Re: Show HN: Saf – simple, reliable, rsync-based, battle tested, rounded backup

#22

Post by dspillett »

BoppreH wrote: post #11

How do you automate the checking if the backup worked correctly, in face of saf bugs, rsync bugs/misconfiguration, or bit rot? My solution is to pick a few random files (plus whatever is new), and compute their hashes on both local and remote versions. But it's slow and probabilistic. ZFS also helps, but I feel it's too transparent to rely on (what if the remote storage changes filesystem).

> But it's slow and probabilistic.

A couple of things I do:

1. Generate a list of files on both sides and the sizes & dates, and compare that ignoring any that have changed/appeared since before the last backup cycle started. Unless your backups are truly massive in terms of number of files this is practical to automate and run at least as often as your backup cycle, and this catches many system errors or simple failures of the backups to run at all.

2. Occasionally checksum the whole damn lot in your latest snapshot and the originals. This can take a lot of time (and expense of you are using child storage with read access charges) so you want to do it less often but it catches bit rot and similar issues. Again you have to skip files that have been touched since the start of the last backup cycle.

3. If you keep a checksum (or list of files with checksum) of each snapshot, occasionally pick one and verify it from scratch. As with hashing the latest snapshot this can be quite resource intensive for massive backups but is fine for mine. You can also just compare meta-data (files, sizes, dates) to a stored list which will catch some types of filesystem corruption affecting your older snapshots.

One of these days I'll might get around to tidy+documenting+publishing my scripts that run all this…

Re: Show HN: Saf – simple, reliable, rsync-based, battle tested, rounded backup

#23

Post by BoppreH »

dspillett wrote: post #22
BoppreH wrote: post #11

How do you automate the checking if the backup worked correctly, in face of saf bugs, rsync bugs/misconfiguration, or bit rot? My solution is to pick a few random files (plus whatever is new), and compute their hashes on both local and remote versions. But it's slow and probabilistic. ZFS also helps, but I feel it's too transparent to rely on (what if the remote storage changes filesystem).

> But it's slow and probabilistic. A couple of things I do: 1. Generate a list of files on both sides and the sizes & dates, and compare that ignoring any that have changed/appeared since before the last backup cycle started. Unless your backups are truly massive in terms of number of files this is practical to automate and run at least as often as your backup cycle, and this catches many system errors or simple fail…

That's close to what I do[1]. The size and date comparison is done by rsync, and I keep a text file with all expected file hashes, so if there's any disagreement between copies I know which one to trust.

These hashes are also ordered so that the top files haven't been checked the longest; part of the script is to take the top N files, checksum them, and move them to the bottom of the list. This guarantees every file is checksum once per N days.

I also donwload a random file in every run, to make sure the connection is not broken.

My use case is personal photos and videos, so I also make sure that my local files are never changed.

And finally, I highly recommend Hetzner Storage Boxes. Not only are they dirty cheap while still giving you ZFS and samba access, you can actually SSH into the box and run simple commands on the files locally, like sha25sum, without paying for network transfers.

[1] https://github.com/boppreh/cloud_backup_script/

Re: Show HN: Saf – simple, reliable, rsync-based, battle tested, rounded backup

#24

Post by otterpro »

Wow, I like this a lot, as it looks easy to run and it can sync to multiple targets. My local backup consists of JBOD (not RAID, ZFS or BTRFS) so I think this should work nicely. I've been using a shell script for doing something similar for backup, but it lacked a lot of the features.

Re: Show HN: Saf – simple, reliable, rsync-based, battle tested, rounded backup

#25

Post by bandyaboot »

tetris11 wrote: post #6
sureglymop wrote: post #4

Have a look at restic for a good alternative to this.

Resric is great but the lack of empty passwords, and the response by the developer about it is very grating: https://github.com/restic/restic/issues/1786

He very politely said he thinks it’s better to keep the password requirement in place and was deciding to do that. What’s grating about that? Personally, I think his concern about users mistakenly not setting a password could be alleviated with an explicit —insecure flag, or similar.

Re: Show HN: Saf – simple, reliable, rsync-based, battle tested, rounded backup

#26

Post by aborsy »

tetris11 wrote: post #6
sureglymop wrote: post #4

Have a look at restic for a good alternative to this.

Resric is great but the lack of empty passwords, and the response by the developer about it is very grating: https://github.com/restic/restic/issues/1786

It’s a good idea to enforce passwords for security. The features of backups done right are incremental backup, snapshot, deduplication, encryption and compression.
Post reply on HN