Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Duplicity: Encrypted bandwidth-efficient backup

duplicity.us

Post reply on HN
Print view View original

41–50 of 104 posts

Re: Duplicity: Encrypted bandwidth-efficient backup

#41

Post by stavros »

sigio wrote: post #36
longwave wrote: post #4

I used this many, many years ago but switched to Borg[0] about five years ago. Duplicity required full backups with incremental deltas, which meant my backups ended up taking too long and using too much disk space. Borg lets you prune older backups at will, because of chunk tracking and deduplication there is no such thing as an incremental backup. [0] https://www.borgbackup.org/

Agree completely... used duplicity many years ago, but switched to Borg and never looked back. Currently doing borg-backups of quite a lot of systems, many every 6 hours, and some, like my main shell-host every 2 hours. It's quick, tiny and easy... and restores are the easiest, just mount the backup, browse the snapshot, and copy files where needed.

After Borg, I switched to Restic:

https://restic.net/

AFAIK, the only difference is that Restic doesn't require Restic installed on the remote server, so you can efficiently backup to things like S3 or FTP. Other than that, both are fantastic.

Re: Duplicity: Encrypted bandwidth-efficient backup

#42

Post by formerly_proven »

_flux wrote: post #34
EuAndreh wrote: post #32

Earlier quoted context omitted.

> though borgbackup is still unable to back up to the same repo from multiple hosts at the same time Wouldn't that mean that, when using encrypted backups, secrets would have to be shared across multiple clients? If I'm understanding it correctly, it sounds like an anti-feature. Do other backup tools do that?

Yes, it seems to be the case; only the data in the server is encrypted, while the key is shared between clients sharing the same repository. I'm not sure if content-addressed storage is feasible to implement otherwise. Maybe use the hash of the the unencrypted or shared-key-encrypted as the key, and then encrypt the per-block keys with keys of the clients who have the contents would do it. In any case, I'm not aware…

CAS-based backup tools leak metadata like a sieve, so they're generally not the best choice for the most paranoid people, which should probably stick to uncompressed tar archives (or zips, which avoid compressing unrelated files together, which leaks data) padded to full 100 megs or so and then encrypted en bloc.

Re: Duplicity: Encrypted bandwidth-efficient backup

#44

Post by thedanbob »

PSA for anyone else as stupid as me: when doing selective restores be very careful about how you set target_directory. "duplicity restore --path-to-restore some_file source_url ~" does not mean "restore some_file to my home directory", it means "replace my home directory with some_file".

Re: Duplicity: Encrypted bandwidth-efficient backup

#45

Post by radlad »

AgentME wrote: post #38
_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

Restic also works like this, and has the following benefits over Borg: multiple hosts can back up to the same repo, and it supports "dumb" remote file hosts that aren't running Borg like S3 or plain SFTP servers.

I really like restic, and am personally happy to use it via the command line. It's very fast and efficient! However, I do wish there was better tooling / wrappers around it. I'd love to be able to set something simple up on my partner's Macbook.

For example, Pika Backup, and Vorta are popular UIs for Borg of which no equivalent exists for Restic, while Borgmatic seems to be a de-facto standard for profile configuration.

For my own purposes, I've been using a script I found on Github[0] for a while, but it only really supports Backblaze B2 AFAIK.[1] I've been meaning to try autorestic[2] and resticprofile[3] as they are potentially more flexible than the script I'm currently using but the fact that there are so many competing tools - many of which are no longer maintained - makes it difficult to choose a specific one.

Prestic[4] looks intriguing for my partner's use, although it seems to have very few users. :\ A fork of Vorta[5] seems to have fizzled out six years ago.

[0] https://github.com/erikw/restic-automatic-backup-scheduler

[1] https://github.com/erikw/restic-automatic-backup-scheduler/i...

[2] https://github.com/cupcakearmy/autorestic

[3] https://github.com/creativeprojects/resticprofile

[4] https://github.com/ducalex/prestic

[5] https://github.com/Mebus/restatic

Re: Duplicity: Encrypted bandwidth-efficient backup

#47

Post by JoshTriplett »

AgentME wrote: post #38
_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

Restic also works like this, and has the following benefits over Borg: multiple hosts can back up to the same repo, and it supports "dumb" remote file hosts that aren't running Borg like S3 or plain SFTP servers.

I'm a huge fan of restic as well. My only complaint is performance and memory usage. I'm looking forward to being able to use Rustic: https://rustic.cli.rs/

Re: Duplicity: Encrypted bandwidth-efficient backup

#48

Post by nijave »

_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

>borgbackup is still unable to back up to the same repo from multiple hosts at the same time,

Basically still an issue. The machine takes an exclusive lock and it also adds override since each machine has to update it's local data cache (or whatever it's called) because they're constantly getting out of sync when another machine backs up

bupstash looks promising as a close-to-but-more-performant borg alternative but it's still basically alpha quality

It's unfortunate peer-to-peer Crashplan died

Re: Duplicity: Encrypted bandwidth-efficient backup

#49

Post by nijave »

Linux-Fan wrote: post #27
_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

> Content-addressed backups sound something like how git stores data, is that the best way to think about them? I think it is a valid way to consider them. Another option is to think of the backup as a special kind of file system snapshot that manifests itself as real files as opposed to data on a block device. > And if so, what would be the main differences between just committing to a git repo for example? The main…

Also git is not good with binary and sort of bolts on git-lfs as a workaround

Re: Duplicity: Encrypted bandwidth-efficient backup

#50

Post by baal80spam »

darrmit wrote: post #14

I've found restic + rclone to be extremely stable and reliable for this same sort of differential backup. I backup to Backblaze B2 and have also used Google Drive with success, even for 1TB+ of data.

+1 for rclone. What a great piece of software.
Post reply on HN