Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

zsync.moria.org.uk

Post reply on HN
Print view View original

11–20 of 49 posts

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#11

Post by feelin_googley »

takeda wrote: post #8

This reminds me about the demo utility that comes wit rsynclib called rdiff, except they approached this problem in less practical way, although it shows rsynclib better. It works as follows: - Let say you already have file that is older version, or perhaps corrupted, you use rdiff to generate its signature - you go then to the place which contains proper file and use the signature file to generate a patch file - the…

Not that I would know any better but I always saw a user-controlled approach built around rdiff as a better alternative than surrendering files to a non-transparent third party such as Dropbox (who, go figure, used librsync originally).

There is an elegant simplicity to rdiff, IMHO.

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#12

Post by cbhl »

js2 wrote: post #9

2005. So I guess this didn't catch on.

Ubuntu still provides zsync of its installation media: http://cdimage.ubuntu.com/ubuntu/releases/17.04/release/

That said, I'm not sure I know of any other major users of it -- most people just use a .torrent (which similarly has checksums of each piece so you know which pieces need to be downloaded).

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#13

Post by teraflop »

feelin_googley wrote: post #11
takeda wrote: post #8

This reminds me about the demo utility that comes wit rsynclib called rdiff, except they approached this problem in less practical way, although it shows rsynclib better. It works as follows: - Let say you already have file that is older version, or perhaps corrupted, you use rdiff to generate its signature - you go then to the place which contains proper file and use the signature file to generate a patch file - the…

Not that I would know any better but I always saw a user-controlled approach built around rdiff as a better alternative than surrendering files to a non-transparent third party such as Dropbox (who, go figure, used librsync originally). There is an elegant simplicity to rdiff, IMHO.

http://duplicity.nongnu.org/ implements this approach, FWIW.

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#14

Post by bch »

feelin_googley wrote: post #11
takeda wrote: post #8

This reminds me about the demo utility that comes wit rsynclib called rdiff, except they approached this problem in less practical way, although it shows rsynclib better. It works as follows: - Let say you already have file that is older version, or perhaps corrupted, you use rdiff to generate its signature - you go then to the place which contains proper file and use the signature file to generate a patch file - the…

Not that I would know any better but I always saw a user-controlled approach built around rdiff as a better alternative than surrendering files to a non-transparent third party such as Dropbox (who, go figure, used librsync originally). There is an elegant simplicity to rdiff, IMHO.

ISTR they may have started w librsync but even years ago, their (modified, actively developed) lib bore little resemblance to the canonical librsync.

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#15

Post by feelin_googley »

teraflop wrote: post #13
feelin_googley wrote: post #11

Earlier quoted context omitted.

Not that I would know any better but I always saw a user-controlled approach built around rdiff as a better alternative than surrendering files to a non-transparent third party such as Dropbox (who, go figure, used librsync originally). There is an elegant simplicity to rdiff, IMHO.

http://duplicity.nongnu.org/ implements this approach, FWIW.

I am aware of another project called "rdiff-backup", also at nongnu.org:

rdiff-backup backs up one directory to another, possibly over a network. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, hard links, dev files, permissions, uid/gid ownership (if it is running as root), and modification times. Finally, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted.

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#16

Post by IanCal »

bruxis wrote: post #10
deelowe wrote: post #6

Earlier quoted context omitted.

It uses a special file that has to be installed on the server. Perhaps you thought (as did I) that this was supposed to work with any file over http? That does not appear to be the case. http://zsync.moria.org.uk/server

I see, this really threw me off: "Rsync over HTTP — zsync provides transfers that are nearly as efficient as rsync -z or cvsup, without the need to run a special server application. All that is needed is an HTTP/1.1-compliant web server . So it works through firewalls and on shared hosting accounts, and gives less security worries." The other child comment also touches on something that I seem to have skipped over, t…

Interesting note on the site is that these files don't need to be on the original server. Anyone can generate and host them.

> in fact, the .zsync can be generated and offered by a third party, while still leaving most of the download to the original distribution site.

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#18

Post by oliwarner »

deelowe wrote: post #6
bruxis wrote: post #3

This makes the claim that you don't need to run any additional software on the server, other than having an HTTP/1.1 compliant host. How does zsync diff against the local file without downloading the contents from the server?

It uses a special file that has to be installed on the server. Perhaps you thought (as did I) that this was supposed to work with any file over http? That does not appear to be the case. http://zsync.moria.org.uk/server

Installed on? Uploaded to.

It's just a pair of files. The big thing you're trying to transfer and the zsync file that details the content of that first file, to guide the downloader.

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#19

Post by vog »

cbhl wrote: post #12
js2 wrote: post #9

2005. So I guess this didn't catch on.

Ubuntu still provides zsync of its installation media: http://cdimage.ubuntu.com/ubuntu/releases/17.04/release/ That said, I'm not sure I know of any other major users of it -- most people just use a .torrent (which similarly has checksums of each piece so you know which pieces need to be downloaded).

Yes, this seems to have been replaced mostly by bittorrent.

On the other hand, HTTP has the advantage that it works through corporate proxies and that is usually not blocked by over-cautious admins.

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#20

Post by dividuum »

cbhl wrote: post #12
js2 wrote: post #9

2005. So I guess this didn't catch on.

Ubuntu still provides zsync of its installation media: http://cdimage.ubuntu.com/ubuntu/releases/17.04/release/ That said, I'm not sure I know of any other major users of it -- most people just use a .torrent (which similarly has checksums of each piece so you know which pieces need to be downloaded).

Not a major user, but we're using zsync for system updates of our Raspberry Pi based digital signage operating system (https://info-beamer.com/hosted). It's pretty great and offers a few things we couldn't do with bittorrent: Every time we have a new release we put together an install.zip file of everything required (kernel, firmware files, initrd, squashfs). Users can download this file directly and unzip it on their SD card and it will boot our OS. For updates we use a previous (see below) version of our install.zip already available on the device and only download the changes. We then unzip that into a new partition for an A/B booting.

Zsync is awesome as we can specify any number of existing files already available on the device (with the -i command line option) and zsync will try to make use of them to minimize downloads. We really use this feature to our advantage: zsync by default will keep the previous version of a file if it's going to overwrite it. So we have two versions of install.zip on a device. When switching between OS releases (stable / testing...) we can switch back and forth with zero additional downloads as both versions are available and zsync makes use of that. Similarly after a user installs our OS, we just have the unpacked artefacts (kernel, etc.) on the SD. We can quickly recreate an initial version of the install.zip file on the device by seeding the download with those files. It's usually just 500k to construct an initial install.zip file we then later use to minimize all future updates.

Post reply on HN