Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Duplicity: Encrypted bandwidth-efficient backup

duplicity.us

Post reply on HN
Print view View original

71–80 of 104 posts

Re: Duplicity: Encrypted bandwidth-efficient backup

#71

Post by stavros »

pdimitar wrote: post #70
stavros wrote: post #69

Earlier quoted context omitted.

Nothing offhand, but basically it can't know what's on the server without reading it all, and if it can't do that locally, it'll have to do it remotely. At that point, might as well re-upload the whole thing. Its front page hints at this, but there must be details somewhere.

Not to make this an endless thread, but I have been wondering about what's the most rsync-friendly backup on-disk layout. I have found Borg to have less files and directories which I would naively think translates to less checks (and the files are not huge, too). I have tried Kopia and Bupstash as well but they both produce a lot of files and directories, much more than Borg. So I think Borg wins at this but I haven'…

I think the advantage of restic is that you don't need to rsync afterwards, it handles all that for you. Combined with its FUSE backup decryption (it mounts the remote backup as a local filesystem you can restore files from), it's very set-and-forget.

Re: Duplicity: Encrypted bandwidth-efficient backup

#72

Post by pdimitar »

stavros wrote: post #71
pdimitar wrote: post #70

Earlier quoted context omitted.

Not to make this an endless thread, but I have been wondering about what's the most rsync-friendly backup on-disk layout. I have found Borg to have less files and directories which I would naively think translates to less checks (and the files are not huge, too). I have tried Kopia and Bupstash as well but they both produce a lot of files and directories, much more than Borg. So I think Borg wins at this but I haven'…

I think the advantage of restic is that you don't need to rsync afterwards, it handles all that for you. Combined with its FUSE backup decryption (it mounts the remote backup as a local filesystem you can restore files from), it's very set-and-forget.

My problem with Restic was that it did not recognize sub-second timestamps of files. I made test scripts that tested it (and were creating files and directories in a hypothetical backup source, and were also changing the files) but then Restic insisted nothing was changed because the changes were happening too fast.

I modified the scripts to do `sleep 1` between each change but it left a sour taste and I never gave Restic a fair chance. I see a good amount of praise in this thread, I'll definitely revisit it when I get a little free time and energy.

Because yeah, it's not expected you'll make a second backup snapshots <1s after the first one. :D

Re: Duplicity: Encrypted bandwidth-efficient backup

#73

Post by russelg »

rabbitofdeath wrote: post #33
tzs wrote: post #23

Not to be confused with Duplicati [1] or Duplicacy [2]. There are too many backup programs whose names start with 'Duplic'. [1] https://www.duplicati.com/ [2] https://duplicacy.com/

Duplicacy for me has been amazing - I use it to backup all of my machines nightly all consolidated into 1 repo that is copied to B2 and it works amazingly. I've restored plenty and have not had any issues.

I'm curious as to why you took that approach. Why not just straight to B2 from each machine? Is it for a redundant local copy of all the backups? If so that sounds like a good idea since restoring from B2 takes ages just because listing the revisions is hella slow for me...

Re: Duplicity: Encrypted bandwidth-efficient backup

#74

Post by duskwuff »

65 wrote: post #40
res0nat0r wrote: post #29

Earlier quoted context omitted.

The major issue out of the box vs any deduping backup software is that S3 doesnt support any deduplication. If you move or rename a 15GB file you're going to have to completely upload it again and also store a second copy and pay for it until your S3 bucket policy purges the previously uploaded file you've deleted. Also aws s3 sync is much slower since it has to iterate over all of the files to see if their size/time…

It's possible to find probable duplicate files with the S3 CLI based on size and tags - I was working a script to do just that but I haven't finished it yet. Alternatively if you want exact backups of your computer you can use the --delete flag which will delete files in the bucket that aren't in the source. I agree this is not the absolute most optimized solution but it does work quite well for me and is easily exte…

> Theoretically if Borgbackup or Duplicity are backing up to S3 they're using all the same commands as the S3 CLI/SDK.

They are not. Both Borg and Duplicity pack files into compressed, encrypted archives before uploading them to S3; "s3 sync" literally just uploads each file as an object with no additional processing.

Re: Duplicity: Encrypted bandwidth-efficient backup

#75

Post by kiwijamo »

thedanbob wrote: post #44

PSA for anyone else as stupid as me: when doing selective restores be very careful about how you set target_directory. "duplicity restore --path-to-restore some_file source_url ~" does not mean "restore some_file to my home directory", it means "replace my home directory with some_file".

I usually make a folder specifically for restores and target anything there -- this avoids this issue.

Re: Duplicity: Encrypted bandwidth-efficient backup

#76

Post by jiggawatts »

KMag wrote: post #26
mizzao wrote: post #21

Earlier quoted context omitted.

Content-addressed backups sound something like how git stores data, is that the best way to think about them? And if so, what would be the main differences between just committing to a git repo for example?

The "rolling window hashes" from the comment suggests sub-file matching at any offset. (See Bently-McIlroy diff algo/how rsync efficiently finds matches, for example.) I'm not aware that git performs this sort of deduplication. Better yet would be to use a rolling hash to decide where to cut the blocks, and then use a locality-aware hash (SimHash, etc.) to find similar blocks. Perform a topological sort to decide whi…

Microsoft's implementation is called Remote Differential Compression: https://learn.microsoft.com/en-us/previous-versions/windows/...

It's available as a built-in component of Windows, it's just a library with an API.

Essentially the MS RDC protocol is just rsync run twice in a row, with the rsync metadata copied via rsync to compress it further.

Re: Duplicity: Encrypted bandwidth-efficient backup

#77

Post by jszymborski »

AgentME wrote: post #38
_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

Restic also works like this, and has the following benefits over Borg: multiple hosts can back up to the same repo, and it supports "dumb" remote file hosts that aren't running Borg like S3 or plain SFTP servers.

Can someone clarify if Restic dedups and compresses encrypted repos like Borg does? I feel like at some point it couldn't but maybe read that it now can?

I ask because my Borg repo is an order of magnitude smaller because of dedup, so it's essential for me.

Re: Duplicity: Encrypted bandwidth-efficient backup

#78

Post by shaunpud »

longwave wrote: post #4

I used this many, many years ago but switched to Borg[0] about five years ago. Duplicity required full backups with incremental deltas, which meant my backups ended up taking too long and using too much disk space. Borg lets you prune older backups at will, because of chunk tracking and deduplication there is no such thing as an incremental backup. [0] https://www.borgbackup.org/

Works a treat with borgmatic https://torsion.org/borgmatic

Re: Duplicity: Encrypted bandwidth-efficient backup

#79

Post by stavros »

pdimitar wrote: post #72
stavros wrote: post #71

Earlier quoted context omitted.

I think the advantage of restic is that you don't need to rsync afterwards, it handles all that for you. Combined with its FUSE backup decryption (it mounts the remote backup as a local filesystem you can restore files from), it's very set-and-forget.

My problem with Restic was that it did not recognize sub-second timestamps of files. I made test scripts that tested it (and were creating files and directories in a hypothetical backup source, and were also changing the files) but then Restic insisted nothing was changed because the changes were happening too fast. I modified the scripts to do `sleep 1` between each change but it left a sour taste and I never gave R…

I'm going to say that was a bit of a niche usage :P

Re: Duplicity: Encrypted bandwidth-efficient backup

#80

Post by CrendKing »

rkagerer wrote: post #53
_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

For those of us who prefer not to ship to the cloud, have you used Kopia Repository Server and is it any good? Does it run on Windows? The documentation refers to files and directories. Does the software let you take a consistent, point-in-time snapshot of a whole drive (or even multiple volumes), e.g. using something like VSS? Or if you want that have you got to use other software (like Macrium Reflect) to produce a…

I wrote this part of the documentation: https://kopia.io/docs/advanced/actions/#windows-shadow-copy

If you have problem using it, please let me know.

Post reply on HN