Try kopia.io is very good
Duplicity: Encrypted bandwidth-efficient backup
31–40 of 104 posts
Re: Duplicity: Encrypted bandwidth-efficient backup
#32I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…
> though borgbackup is still unable to back up to the same repo from multiple hosts at the same time
Wouldn't that mean that, when using encrypted backups, secrets would have to be shared across multiple clients?
If I'm understanding it correctly, it sounds like an anti-feature. Do other backup tools do that?
Re: Duplicity: Encrypted bandwidth-efficient backup
#33Not to be confused with Duplicati [1] or Duplicacy [2]. There are too many backup programs whose names start with 'Duplic'. [1] https://www.duplicati.com/ [2] https://duplicacy.com/
Duplicacy for me has been amazing - I use it to backup all of my machines nightly all consolidated into 1 repo that is copied to B2 and it works amazingly. I've restored plenty and have not had any issues.
Re: Duplicity: Encrypted bandwidth-efficient backup
#34I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…
> though borgbackup is still unable to back up to the same repo from multiple hosts at the same time Wouldn't that mean that, when using encrypted backups, secrets would have to be shared across multiple clients? If I'm understanding it correctly, it sounds like an anti-feature. Do other backup tools do that?
Yes, it seems to be the case; only the data in the server is encrypted, while the key is shared between clients sharing the same repository.
I'm not sure if content-addressed storage is feasible to implement otherwise. Maybe use the hash of the the unencrypted or shared-key-encrypted as the key, and then encrypt the per-block keys with keys of the clients who have the contents would do it. In any case, I'm not aware of such backup tools (I imagine most just don't encrypt anything).
Re: Duplicity: Encrypted bandwidth-efficient backup
#35I've found restic + rclone to be extremely stable and reliable for this same sort of differential backup. I backup to Backblaze B2 and have also used Google Drive with success, even for 1TB+ of data.
I agree. restic with it's simple ability to mount (using fusefs) a backup so you can just copy out the file(s) you need, is so wonderful. A single binary that you just download and can SCP around the place to any device etc.
It's fantastic to have so many great open source backup solutions. I investigated many and settled on restic. It still brings me joy to actually use it, it's so simple and hassle free.
Re: Duplicity: Encrypted bandwidth-efficient backup
#36I used this many, many years ago but switched to Borg[0] about five years ago. Duplicity required full backups with incremental deltas, which meant my backups ended up taking too long and using too much disk space. Borg lets you prune older backups at will, because of chunk tracking and deduplication there is no such thing as an incremental backup. [0] https://www.borgbackup.org/
Agree completely... used duplicity many years ago, but switched to Borg and never looked back. Currently doing borg-backups of quite a lot of systems, many every 6 hours, and some, like my main shell-host every 2 hours.
It's quick, tiny and easy... and restores are the easiest, just mount the backup, browse the snapshot, and copy files where needed.
Re: Duplicity: Encrypted bandwidth-efficient backup
#37Excellent piece of software, and relatively simple to use with gpg encryption. I've been using it for many years. My only complaint is that, like a lot of software written in Python, it has no regard for traditional UNIX behavior (keep quiet unless you have something meaningful to say), so I have to live with cron reporting stuff like: "/usr/lib/python2.7/dist-packages/paramiko/rsakey.py:99: DeprecationWarning: signe…
You can try setting `export PYTHONWARNINGS="ignore"` to suppress warnings.
Re: Duplicity: Encrypted bandwidth-efficient backup
#38I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…
Restic also works like this, and has the following benefits over Borg: multiple hosts can back up to the same repo, and it supports "dumb" remote file hosts that aren't running Borg like S3 or plain SFTP servers.
Re: Duplicity: Encrypted bandwidth-efficient backup
#39Not to be confused with Duplicati [1] or Duplicacy [2]. There are too many backup programs whose names start with 'Duplic'. [1] https://www.duplicati.com/ [2] https://duplicacy.com/
While we're on the topic of Duplicati, I feel the need to share my personal experience; one that's echoed by lots of folks online. Duplicati restores can take what seems like the heat death of the universe to restore a repo as little as 500Gb. I've lost a laptop worth of files to it. You can find tonnes of posts on the Duplicati forums which retell the same story [0]. I've moved to Borg and backing up to a Hetzner St…
> Remember folks, test your backups.
Since you mention it, I am seizing the opportunity to ask: how should borg backup be tested ? Can it be automated ?
Re: Duplicity: Encrypted bandwidth-efficient backup
#40Earlier quoted context omitted.
S3 has bucket versioning if you want to have multiple backups. The S3 sync command also does differential backups; if you for example try to run the script over and over it will only upload new/different files.
The major issue out of the box vs any deduping backup software is that S3 doesnt support any deduplication. If you move or rename a 15GB file you're going to have to completely upload it again and also store a second copy and pay for it until your S3 bucket policy purges the previously uploaded file you've deleted. Also aws s3 sync is much slower since it has to iterate over all of the files to see if their size/time…
It's possible to find probable duplicate files with the S3 CLI based on size and tags - I was working a script to do just that but I haven't finished it yet. Alternatively if you want exact backups of your computer you can use the --delete flag which will delete files in the bucket that aren't in the source.
I agree this is not the absolute most optimized solution but it does work quite well for me and is easily extendible with other scripts and S3 CLI commands. Theoretically if Borgbackup or Duplicity are backing up to S3 they're using all the same commands as the S3 CLI/SDK.
Besides, shell scripting is fun!