Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A performance comparison of Duplicacy, restic, Attic, and duplicity

github.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: A performance comparison of Duplicacy, restic, Attic, and duplicity

#41

Post by acrosync »

comice wrote: post #35
acrosync wrote: post #30

Earlier quoted context omitted.

Free or not free aside, my question is, does it matter to personal users between this free-for-personal-use license and any of those more permissive licenses like MIT, BSD, or GPL?

It matters to me as a personal user because my use of duplicacy might change at some point and suddenly I'd lose rights to use it (unless I pay). I'd lose rights to any development contributions I might have made unless I pay. And as a personal user, I can't use any code from Duplicacy in any other project. I can't even, say, create a package for it and get it included in Debian. And aside from some of these practica…

Thanks for your feedback. The reason I don't like open-source licenses is that I don't want for-profit companies to use my software without paying. The ideal license would be the one that requires them to pay while being appealing to personal users like you. I don't think these two goals are irreconcilable, but unfortunately such a license doesn't exist yet.

Re: A performance comparison of Duplicacy, restic, Attic, and duplicity

#43

Post by acrosync »

_JamesA_ wrote: post #40
Theriault wrote: post #36

Earlier quoted context omitted.

It's stopped me from using it for personal use. From a practical standpoint, it makes it more difficult for me to trust that it will be maintained in the long-term, or that I can extend its functionality if I see a need. Ideologically, I'm somewhat uncomfortable using duplicacy when fully-free alternatives exist. I'm not a free software purist by any definition (I use steam. I have a Netflix subscription. My android…

It is an issue of trust for something as important as a backup tool. Duplicacy sounds great but it has 6 [0] contributors and Borg has 107 [1]. It's obvious which one has more eyes on it. Plus I can apt-get install borgbackup / apt-get upgrade which adds another level of trust. [0]: https://github.com/gilbertchen/duplicacy [1]: https://github.com/borgbackup/borg

We just released the source code less than 2 months ago. The difference in contributor counts may not be this big in one or two years from now.

Re: A performance comparison of Duplicacy, restic, Attic, and duplicity

#44

Post by comice »

acrosync wrote: post #41
comice wrote: post #35

Earlier quoted context omitted.

It matters to me as a personal user because my use of duplicacy might change at some point and suddenly I'd lose rights to use it (unless I pay). I'd lose rights to any development contributions I might have made unless I pay. And as a personal user, I can't use any code from Duplicacy in any other project. I can't even, say, create a package for it and get it included in Debian. And aside from some of these practica…

Thanks for your feedback. The reason I don't like open-source licenses is that I don't want for-profit companies to use my software without paying. The ideal license would be the one that requires them to pay while being appealing to personal users like you. I don't think these two goals are irreconcilable, but unfortunately such a license doesn't exist yet.

I did wonder if being fully free might encourage more users who might fund you in other ways but Borg backup isn't making very much like that, so perhaps not: https://www.bountysource.com/teams/borgbackup

The AGPL license might be a step in the right direction (for your requirements). It aims to at least ensure that if companies use the code to provide a service to other users, they have to release their changes. You can sell those companies a different license if they don't want to accept the AGPL (you'd have to have a contributor agreement to assign copyright to you though, to allow you to relicense code at your discretion like that).

Or there is the open core model (like nginx-plus), where you provide the code under an open source license but provide some additional "enterprise" features (like your vmware stuff) to only those that pay. I'm not a fan but it seems to work for some.

Anyway, duplicacy sounds a great design. All the best with it!

Re: A performance comparison of Duplicacy, restic, Attic, and duplicity

#46

Post by tatersolid »

willvarfar wrote: post #32
ams6110 wrote: post #27

> duplicity has a serious flaw in its incremental model -- the user has to decide whether to perform a full backup or an incremental backup on each run. That is because while an incremental backup saves a lot of storage space, it is also dependent on previous backups due to the design of duplicity, making it impossible to delete any single backup on a long chain of dependent backups. So there is always a dilemma of h…

Classic source control had this problem. The clever trick is to reencode the previous most-recent backup as a delta from the current state, and do a full-backup of the current state, rather than encoding each new backup as a delta from the previous state (which becomes slower and slower to compute, the more previous states you have). Problem solved :)

That's really expensive when your previous backup is on cloud storage

Re: A performance comparison of Duplicacy, restic, Attic, and duplicity

#47

Post by willvarfar »

tatersolid wrote: post #46
willvarfar wrote: post #32

Earlier quoted context omitted.

Classic source control had this problem. The clever trick is to reencode the previous most-recent backup as a delta from the current state, and do a full-backup of the current state, rather than encoding each new backup as a delta from the previous state (which becomes slower and slower to compute, the more previous states you have). Problem solved :)

That's really expensive when your previous backup is on cloud storage

To compute a delta, you need the previous version. If this previous version is computed from a single file - the previous snapshot - then that's actually less data and effort than if its computed by taking an old snapshot and replaying all the deltas upto the current time.

Re: A performance comparison of Duplicacy, restic, Attic, and duplicity

#48

Post by dom0 »

acrosync wrote: post #26
tombrossman wrote: post #23

Earlier quoted context omitted.

You realize the name Borg comes from the Attic author, Jonas Borgström, right? No one else calls it Attic as the two are different projects.

I noticed that, but didn't know if Borg has another meaning. I can understand why they forked the project, but in my opinion a name that makes the origin more obvious would have been better.

"Borg" was chosen, because it emphasizes collaborative development — and because someone is a Star Trek fan ;)

Re: A performance comparison of Duplicacy, restic, Attic, and duplicity

#49

Post by dom0 »

e12e wrote: post #39

It's a little odd to not benchmark backup over the network - a backup taken to the same physical disk as the source of the data isn't very useful - for that use-case taking a filesystem snapshot[s] would probably be faster and more useful. Perhaps in combination with a checksumming tool, like [c], or with a filesystem like ZFS. Also, it can be difficult in a lot of environments to sustain more than 100mbps write to a…

> All that said, it's interesting to see that a) duplicity seems slow, and b) very consistent in terms of speed. I wonder if there's some low-hanging fruit for optimization there.

Duplicity is classic delta-backup. It always reads all files and calculates a delta to a different version of the file, hence the fairly consistent performance. Performance of deduplicating archivers is more difficult to predict.

Re: A performance comparison of Duplicacy, restic, Attic, and duplicity

#50

Post by dom0 »

My two (possibly biased, much like the author's) cents.

- No network-based tests; e.g. a typical fast internet connection (say 100/40 or 50/20 MBit/s) with a few dozen ms latency to some server or cloud service. This is of course difficult because these tend to be bad on reproducibility. For a network-based test, not only time is interesting, but total RX/TX as well.

- I'm really surprised at restic's performance. It uses far more CPU than Borg in almost all tests... and Borg is already notoriously inefficient in it's CPU usage when looking at object throughput (restic: "fast, efficient"?). I don't mean to bash, I'm just surprised.

- restic's deduplication performance might hint at Rabin Fingerprints being worse than Buzhash, but there might be other issue(s) leading to this result.

- Besides CPU time, memory (peak) usage would be interesting.

> For instance, file hashes enable users to quickly identify which files in existing backups are changed. They also allow third-party tools to compare files on disks to those in the backups.

To be fair, Borg can calculate a variety of file hashes (MD5, SHA1, SHA2, ...) on the fly with "borg list". There are "borg diff" (to compare two archives) and "borg mount -o versions" as well, though the latter is generally impractical for looking at a large number of archives.

> Again, by not computing the file hash helped improve the performance, but at the risk of possible undetected data corruption.

I can't deduce how the last part follows (", but..."). Care to explain?

Post reply on HN