Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Duplicity: Encrypted bandwidth-efficient backup

duplicity.us

Post reply on HN
Print view View original

21–30 of 104 posts

Re: Duplicity: Encrypted bandwidth-efficient backup

#21

Post by mizzao »

_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

Content-addressed backups sound something like how git stores data, is that the best way to think about them?

And if so, what would be the main differences between just committing to a git repo for example?

Re: Duplicity: Encrypted bandwidth-efficient backup

#22

Post by tussa »

longwave wrote: post #4

I used this many, many years ago but switched to Borg[0] about five years ago. Duplicity required full backups with incremental deltas, which meant my backups ended up taking too long and using too much disk space. Borg lets you prune older backups at will, because of chunk tracking and deduplication there is no such thing as an incremental backup. [0] https://www.borgbackup.org/

I did the same. I had some weird path issues with Duplicity.

Borg is now my holy backup grail. Wish I could backup incrementally to AWS glacier storage but that just me sounding like an ungrateful begger. I'm incredibly grateful and happy with Borg!

Re: Duplicity: Encrypted bandwidth-efficient backup

#24

Post by monkey26 »

darrmit wrote: post #14

I've found restic + rclone to be extremely stable and reliable for this same sort of differential backup. I backup to Backblaze B2 and have also used Google Drive with success, even for 1TB+ of data.

I've been using Restic since 2017 without issue. Tried Kopia for a while, but its backup size ballooned on me, maybe it wasn't quite ready.

Re: Duplicity: Encrypted bandwidth-efficient backup

#26

Post by KMag »

mizzao wrote: post #21
_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

Content-addressed backups sound something like how git stores data, is that the best way to think about them? And if so, what would be the main differences between just committing to a git repo for example?

The "rolling window hashes" from the comment suggests sub-file matching at any offset. (See Bently-McIlroy diff algo/how rsync efficiently finds matches, for example.) I'm not aware that git performs this sort of deduplication.

Better yet would be to use a rolling hash to decide where to cut the blocks, and then use a locality-aware hash (SimHash, etc.) to find similar blocks. Perform a topological sort to decide which blocks to store as diffs of others.

Microsoft had some enterprise product that performed distribution somewhat like this, but also recursively using similarity hashes to see if the diffs were similar to existing files on the far machine.

Re: Duplicity: Encrypted bandwidth-efficient backup

#27

Post by Linux-Fan »

_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

> Content-addressed backups sound something like how git stores data, is that the best way to think about them?

I think it is a valid way to consider them. Another option is to think of the backup as a special kind of file system snapshot that manifests itself as real files as opposed to data on a block device.

> And if so, what would be the main differences between just committing to a git repo for example?

The main difference is that good backup tools allow you to delete backups and free up the space whereas git is not really designed for this.

Re: Duplicity: Encrypted bandwidth-efficient backup

#28

Post by tmalsburg2 »

darrmit wrote: post #14

I've found restic + rclone to be extremely stable and reliable for this same sort of differential backup. I backup to Backblaze B2 and have also used Google Drive with success, even for 1TB+ of data.

+1 for restic. I tried various solutions and restic is the best by far. So fast, so reliable.

https://restic.net/

Re: Duplicity: Encrypted bandwidth-efficient backup

#29

Post by res0nat0r »

65 wrote: post #11
rakoo wrote: post #8

Earlier quoted context omitted.

Your script doesn't do the same thing as duplicity. Your script mirrors the local directory with your bucket. It loses all history. Duplicity does backups (ie with history) but not just that, it does differential backups to not upload everything all the time.

S3 has bucket versioning if you want to have multiple backups. The S3 sync command also does differential backups; if you for example try to run the script over and over it will only upload new/different files.

The major issue out of the box vs any deduping backup software is that S3 doesnt support any deduplication. If you move or rename a 15GB file you're going to have to completely upload it again and also store a second copy and pay for it until your S3 bucket policy purges the previously uploaded file you've deleted. Also aws s3 sync is much slower since it has to iterate over all of the files to see if their size/timestamp has changed. Something like borgbackup is much faster as it uses smarter caching to skip unchanged directories etc.

Re: Duplicity: Encrypted bandwidth-efficient backup

#30

Post by jszymborski »

tzs wrote: post #23

Not to be confused with Duplicati [1] or Duplicacy [2]. There are too many backup programs whose names start with 'Duplic'. [1] https://www.duplicati.com/ [2] https://duplicacy.com/

While we're on the topic of Duplicati, I feel the need to share my personal experience; one that's echoed by lots of folks online.

Duplicati restores can take what seems like the heat death of the universe to restore a repo as little as 500Gb. I've lost a laptop worth of files to it. You can find tonnes of posts on the Duplicati forums which retell the same story [0].

I've moved to Borg and backing up to a Hetzner Storage Box. I've restored many times with no issue.

Remember folks, test your backups.

[0] https://forum.duplicati.com/t/several-days-and-no-restore-fe...

Post reply on HN