Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Duplicity: Encrypted bandwidth-efficient backup

duplicity.us

Post reply on HN
Print view View original

91–100 of 104 posts

Re: Duplicity: Encrypted bandwidth-efficient backup

#91

Post by flagged24 »

jszymborski wrote: post #30
tzs wrote: post #23

Not to be confused with Duplicati [1] or Duplicacy [2]. There are too many backup programs whose names start with 'Duplic'. [1] https://www.duplicati.com/ [2] https://duplicacy.com/

While we're on the topic of Duplicati, I feel the need to share my personal experience; one that's echoed by lots of folks online. Duplicati restores can take what seems like the heat death of the universe to restore a repo as little as 500Gb. I've lost a laptop worth of files to it. You can find tonnes of posts on the Duplicati forums which retell the same story [0]. I've moved to Borg and backing up to a Hetzner St…

Same story, Hetzner storage box, local backups and at times a backup to my Android phone. Automated backup testing and a notification on Telegram if anything is amiss.

Re: Duplicity: Encrypted bandwidth-efficient backup

#92

Post by KMag »

jiggawatts wrote: post #76
KMag wrote: post #26

Earlier quoted context omitted.

The "rolling window hashes" from the comment suggests sub-file matching at any offset. (See Bently-McIlroy diff algo/how rsync efficiently finds matches, for example.) I'm not aware that git performs this sort of deduplication. Better yet would be to use a rolling hash to decide where to cut the blocks, and then use a locality-aware hash (SimHash, etc.) to find similar blocks. Perform a topological sort to decide whi…

Microsoft's implementation is called Remote Differential Compression: https://learn.microsoft.com/en-us/previous-versions/windows/... It's available as a built-in component of Windows, it's just a library with an API. Essentially the MS RDC protocol is just rsync run twice in a row, with the rsync metadata copied via rsync to compress it further.

> Essentially the MS RDC protocol is just rsync run twice in a row, with the rsync metadata copied via rsync to compress it further.

There's an important difference is that RDC uses a locality-sensive hash algorithm (MinHash, IIRC) to find files likely to have matching sections, whereas rsync only considers the version of the same file sitting on the far host. rsync encodes differences on each file in isolation, whereas RDC looks at the entire corpus of files on the volume.

For example, if you do the Windows equivalent of cat local/b.txt >> local/a.txt, rsync is going to miss the opportunity to encode local/a.txt -> remote/a.txt using matching runs from a common local/b.txt and remote/b.txt. However, RDC has the opportunity to notice that the local/a.txt -> remote/a.txt diff is very similar to remote/b.txt and further delta-encode the diff as a diff against remote/b.txt.

Re: Duplicity: Encrypted bandwidth-efficient backup

#93

Post by abadpoli »

stavros wrote: post #69
pdimitar wrote: post #68

Earlier quoted context omitted.

That I can't really speak of. I know it does not reupload the same files at least (uses timestamps) but never really checked about only uploading file diffs. Do you have a direct link I can look at?

Nothing offhand, but basically it can't know what's on the server without reading it all, and if it can't do that locally, it'll have to do it remotely. At that point, might as well re-upload the whole thing. Its front page hints at this, but there must be details somewhere.

I think you’re misunderstanding something. There’s no need, and even no possibility, to have “rclone support” on the server, and also no need to “read it all”. rclone uses the features of whatever storage backend you’re using; if you back up to S3, it uses the content hashes, tags, and timestamps that it gets from bucket List requests, which is the same way that Restic works.

Borg does have the option to run both a client-side and a server-side process if you’re backing up to a remote server over SSH, but it’s entirely optional.

Re: Duplicity: Encrypted bandwidth-efficient backup

#94

Post by stavros »

abadpoli wrote: post #93
stavros wrote: post #69

Earlier quoted context omitted.

Nothing offhand, but basically it can't know what's on the server without reading it all, and if it can't do that locally, it'll have to do it remotely. At that point, might as well re-upload the whole thing. Its front page hints at this, but there must be details somewhere.

I think you’re misunderstanding something. There’s no need, and even no possibility, to have “rclone support” on the server, and also no need to “read it all”. rclone uses the features of whatever storage backend you’re using; if you back up to S3, it uses the content hashes, tags, and timestamps that it gets from bucket List requests, which is the same way that Restic works. Borg does have the option to run both a c…

Ah, you're right, I got confused between rsync and rclone's server-side transfers.

Re: Duplicity: Encrypted bandwidth-efficient backup

#95

Post by GTP »

_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

> their content blobs are shared

Doesn't this increase the chance of data loss? If a blob gets corrupted, then all the backups referencing that blob will have the same corrupted file(s). This is similar to having a corrupted index in an incremental backup chain (or maybe in this case you would lose everything?), but in the case of incremental backups the risk is mitigated by periodically performing full backups. Also my gut feeling is that you will save space with content-addressed backups only if you're backing up multiple machines that share files, but in the tipical average user scenario where one is backing up a single PC you get a similar space usage. Keep in mind that you tipically delete bacups older than a certain threshold. Could you maybe comment on my points?

Re: Duplicity: Encrypted bandwidth-efficient backup

#96

Post by GTP »

longwave wrote: post #4

I used this many, many years ago but switched to Borg[0] about five years ago. Duplicity required full backups with incremental deltas, which meant my backups ended up taking too long and using too much disk space. Borg lets you prune older backups at will, because of chunk tracking and deduplication there is no such thing as an incremental backup. [0] https://www.borgbackup.org/

Also duplicity let's you automatically delete backups older than a certain amount of time, what is the difference?

Re: Duplicity: Encrypted bandwidth-efficient backup

#97

Post by sliken »

GTP wrote: post #95
_flux wrote: post #9

I've moved to using backup tools using content-based ids with rolling window hashes, which allows deduplicating content even between different hosts—and crucially handles moving content from one host to another efficiently—even though in other scenarios I'm guessing rdiff-algorithm can produce smaller backups. The problem I have with duplicity and backups tools of its kind is that you still need to create a full back…

> their content blobs are shared Doesn't this increase the chance of data loss? If a blob gets corrupted, then all the backups referencing that blob will have the same corrupted file(s). This is similar to having a corrupted index in an incremental backup chain (or maybe in this case you would lose everything?), but in the case of incremental backups the risk is mitigated by periodically performing full backups. Also…

Sure you could have multiple level-0 backups to increase the odds of whatever blob you have corrupted can be found in other copies, but that's inefficient.

It's much more efficient to deduplicate, then add redundancy. Like say storing said blobs on a RAIDz3. Or use backblaze's approach and split the blob into 17 pieces, add 3 pieces of redundancy, and distribute the chunks across 20 racks.

If you are serious of course you'd have an onsite backup, deduplicated, with added redundancy AND the same offsite.

Re: Duplicity: Encrypted bandwidth-efficient backup

#99

Post by rabbitofdeath »

russelg wrote: post #73
rabbitofdeath wrote: post #33

Earlier quoted context omitted.

Duplicacy for me has been amazing - I use it to backup all of my machines nightly all consolidated into 1 repo that is copied to B2 and it works amazingly. I've restored plenty and have not had any issues.

I'm curious as to why you took that approach. Why not just straight to B2 from each machine? Is it for a redundant local copy of all the backups? If so that sounds like a good idea since restoring from B2 takes ages just because listing the revisions is hella slow for me...

I like the one-repository approach as it works well - it resides on my NAS which is always on and it is a local backup for multiple machines. From there, I replicate to B2 as an emergency copy in the event my NAS (RAID is not backup!) dies or something horrific. I tried to make it future-proof as much as possible too - with Duplicacy, you can easily clone/copy repositories so new hardware migration will be extremely simple. Not only simple with local hardware, but easy to migrate to a different cloud storage as well. Just don't lose your repository encryption key!

Re: Duplicity: Encrypted bandwidth-efficient backup

#100

Post by pengaru »

sliken wrote: post #98
pengaru wrote: post #85

Whatever you do, don't use rdiff-backup.

Why? I've had no issues.

For starters it has a tendency to paint itself into a corner on ENOSPC situations. You won't even be able to perform a restore if a backup was started but unfinished because it ran out of space. There's this process of "regressing" the repo [0] which must occur before you can do practically anything after an interrupted/failed backup. What this actually must do is undo the partial forward progress, by performing what's effectively a restore of the files that got pushed into the future relative to the rest of the repository, which requires more space for any actually modified files. Unless you have/can create free space to do these things, it can become wedged... and if it's a dedicated backup system where you've intentionally filled disks up with restore points, you can find yourself having to throw out backups just to make things functional again.

That's the most obvious glaring problem, beyond that it's just kind of garbage in terms of the amount of space and time it requires to perform restores. Especially restores of files having many reverse-differential increments leading back to the desired restore point. It can require ~2X a given file's size in spare space to assemble the desired version, while it iteratively reconstructs all the intermediate versions in arriving at the desired version. Unless someone improved this since I last had to deal with it, which is possible, it's been years.

Source: Ages ago I worked for a startup[1] that shipped a backup appliance originally implemented by contractors using rdiff-backup behind the scenes. Writing a replacement that didn't suck but was compatible with rdiff-backup's repos while adding newfangled stuff like transactional backups with no need for "regress", direct read-only FUSE access of restore points without needing space, and synthetic virtual-NTFS style access for booting VMs off restore points consumed several years of my life...

There are far better options in 2024.

[0] https://github.com/rdiff-backup/rdiff-backup/blob/master/src...

[1] https://www.crunchbase.com/organization/axcient

Post reply on HN