We are using this for addon synchronization in our community through Arma3Sync. On server side we need to "build" repository - it just generates .zsync files, then clients are downloading just diff. Update size came down from 10-15GB to <3GB.
Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)
21–30 of 49 posts
Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)
#222005. So I guess this didn't catch on.
Ubuntu still provides zsync of its installation media: http://cdimage.ubuntu.com/ubuntu/releases/17.04/release/ That said, I'm not sure I know of any other major users of it -- most people just use a .torrent (which similarly has checksums of each piece so you know which pieces need to be downloaded).
A long time ago on a crappy restricted internet I used this and jigdo at different times to download the Ubuntu ISO. Being able to do so for Ubuntu and not other distributions was another factor in my using Ubuntu back then.
Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)
#23Earlier quoted context omitted.
http://duplicity.nongnu.org/ implements this approach, FWIW.
I am aware of another project called "rdiff-backup", also at nongnu.org: rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental bac…
Yes! I use this and like that it's so simple, and that the latest version of the backup is easily available as plain files. (Any metadata that the filesystem doesn't support are stored separately in files, so it works across different type of filesystems and operating systems.) There is even a FUSE filesystem, "rdiff-backup-fs", for mounting the whole backup history, with each backup point in a subdirectory of its own, like it should be!
Unfortunately, it seems not to be developed any longer, and it has a few things that would need ironing out:
* You can't pause a backup and continue later. * Some operations (notably recovery after an aborted backup run) is excruciatingly slow. It takes tens of hours for me with a backup of 40 GB or so (on a low-powered computer as server, though). I think rdiff-backup-fs is resource hungry as well, which is perhaps partly understandable, since it has to go through a series of reverse diffs to present old versions of a file. * I tried it on Windows once, and it could apparently not handle paths longer than a few hundred characters (due to using that older Windows API, whatever it's called). * You can't delete intermediate backups, only the oldest one.
Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)
#24I've been thinking that we should replace "cp" with "scp", and then replace both of them with "rsync". Is that a bad idea?
Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)
#25There's also xdelta which is just an algorithm / program for calculating and applying binary diffs. I suppose the advantage of zsync is that you can always point "new" users to "/current.file", and use zsync to patch "up" to the latest version - with xdelta people would need to explicitly get "my..current.xdelta".
Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)
#26There's also xdelta which is just an algorithm / program for calculating and applying binary diffs. I suppose the advantage of zsync is that you can always point "new" users to "/current.file", and use zsync to patch "up" to the latest version - with xdelta people would need to explicitly get "my..current.xdelta". http://xdelta.org/
The big difference is that xdelta (much like any diff program) needs both the old and the new versions to create a patch. With zsync, the server only needs to have the new version (which is the only one of interest), and the clients then gets only the parts they want, because they only have the old version.
zsync also does all the fetching stuff directly.
Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)
#27Earlier quoted context omitted.
Ubuntu still provides zsync of its installation media: http://cdimage.ubuntu.com/ubuntu/releases/17.04/release/ That said, I'm not sure I know of any other major users of it -- most people just use a .torrent (which similarly has checksums of each piece so you know which pieces need to be downloaded).
Yes, this seems to have been replaced mostly by bittorrent. On the other hand, HTTP has the advantage that it works through corporate proxies and that is usually not blocked by over-cautious admins.
I'll note that bittorrent uses HTTP for a few different things:
- HTTP(s) based trackers (although UDP is more common these days) - HTTP webseeds/mirrors (BEP-17) - (if you count it) webtorrent uses Websocket trackers and can support HTTP webseeds (although really of course the P2P is WebRTC)
Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)
#28I've been thinking that we should replace "cp" with "scp", and then replace both of them with "rsync". Is that a bad idea?
For what use case? We do use `rsync` instead of `cp` in some capacity, even for local-to-local file copy - as there is slightly more verification of a successful copy, and the destination is a quirky flash medium. Not sure how SCP would help here.
Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)
#29I've been thinking that we should replace "cp" with "scp", and then replace both of them with "rsync". Is that a bad idea?
For what use case ? We do use `rsync` instead of `cp` in some capacity, even for local-to-local file copy - as there is slightly more verification of a successful copy, and the destination is a quirky flash medium. Not sure how SCP would help here.
For all use cases.
> Not sure how SCP would help here.
I mean that my train of though was first "scp, it seems, can do everything cp can, and more, so let's drop cp and use scp instead", then "but hey, rsync, it seems, can do everything that scp can, and more, so let's drop scp too and use rsync instead".
Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)
#30Earlier quoted context omitted.
I am aware of another project called "rdiff-backup", also at nongnu.org: rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental bac…
Yes! I use this and like that it's so simple, and that the latest version of the backup is easily available as plain files. (Any metadata that the filesystem doesn't support are stored separately in files, so it works across different type of filesystems and operating systems.) There is even a FUSE filesystem, "rdiff-backup-fs", for mounting the whole backup history, with each backup point in a subdirectory of its ow…
Have you tried rsnapshot?