Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

zsync.moria.org.uk

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#31

Post by Piskvorrr »

Bromskloss wrote: post #29
Piskvorrr wrote: post #28

Earlier quoted context omitted.

For what use case ? We do use `rsync` instead of `cp` in some capacity, even for local-to-local file copy - as there is slightly more verification of a successful copy, and the destination is a quirky flash medium. Not sure how SCP would help here.

For all use cases. > Not sure how SCP would help here. I mean that my train of though was first "scp, it seems, can do everything cp can, and more, so let's drop cp and use scp instead", then "but hey, rsync, it seems, can do everything that scp can, and more, so let's drop scp too and use rsync instead".

Possibly. `cp` is ancient and rather basic; OTOH, it is everywhere (as opposed to `rsync`, which I found out the hard way) and it is tiny (fewer toggles to push - less stuff to break).

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#32

Post by iFire »

I'm not related to Itchio.

If you are looking for a maintained system for online systems that provide software updates I would look into https://github.com/itchio/wharf-spec.

Wharf is used for Itchio to sync folder structures differentially / incrementally. It uses the latest compression algorithms. It has a reference server.

Alternatively, a port of zsync is https://github.com/salesforce/zsync4j written in Java.

I had trouble compiling zsync for windows.

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#33

Post by alex_hirner »

dividuum wrote: post #20
cbhl wrote: post #12

Earlier quoted context omitted.

Ubuntu still provides zsync of its installation media: http://cdimage.ubuntu.com/ubuntu/releases/17.04/release/ That said, I'm not sure I know of any other major users of it -- most people just use a .torrent (which similarly has checksums of each piece so you know which pieces need to be downloaded).

Not a major user, but we're using zsync for system updates of our Raspberry Pi based digital signage operating system ( https://info-beamer.com/hosted ). It's pretty great and offers a few things we couldn't do with bittorrent: Every time we have a new release we put together an install.zip file of everything required (kernel, firmware files, initrd, squashfs). Users can download this file directly and unzip it on th…

Did any IoT device management service fit the bill too (back then or now)? We are heading towards a similar use case.

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#34

Post by pmoriarty »

cbhl wrote: post #12
js2 wrote: post #9

2005. So I guess this didn't catch on.

Ubuntu still provides zsync of its installation media: http://cdimage.ubuntu.com/ubuntu/releases/17.04/release/ That said, I'm not sure I know of any other major users of it -- most people just use a .torrent (which similarly has checksums of each piece so you know which pieces need to be downloaded).

The AntiX Linux distro also provides zsync downloads:

https://sourceforge.net/projects/antix-linux/files/Final/ant...

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#35

Post by blacksqr »

Fossil (https://www.fossil-scm.org) is a SCM tool that uses the rsync algorithm for syncing repositories. It has a built-in web server, and can also be accessed via CGI from any CGI-capable web server. It also has a SSH option.

Its features make it very handy for a number of file transfer/sync tasks, over and above its chief SCM role.

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#36

Post by rripken »

IanCal wrote: post #16
bruxis wrote: post #10

Earlier quoted context omitted.

I see, this really threw me off: "Rsync over HTTP — zsync provides transfers that are nearly as efficient as rsync -z or cvsup, without the need to run a special server application. All that is needed is an HTTP/1.1-compliant web server . So it works through firewalls and on shared hosting accounts, and gives less security worries." The other child comment also touches on something that I seem to have skipped over, t…

Interesting note on the site is that these files don't need to be on the original server. Anyone can generate and host them. > in fact, the .zsync can be generated and offered by a third party, while still leaving most of the download to the original distribution site.

Interesting. How would that work for clients? If the destination file already exists on the client look for a .zsync on the hosting server, if its not there look for one at https://thirdpartyzsyncs.com?url=someurl ? What happens if the .zsync is out of sync with the resource?

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#37

Post by eeZah7Ux »

iFire wrote: post #32

I'm not related to Itchio. If you are looking for a maintained system for online systems that provide software updates I would look into https://github.com/itchio/wharf-spec . Wharf is used for Itchio to sync folder structures differentially / incrementally. It uses the latest compression algorithms. It has a reference server. Alternatively, a port of zsync is https://github.com/salesforce/zsync4j written in Java. I…

How is the wharf protocol better than zsync?

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#38

Post by eeZah7Ux »

cbhl wrote: post #12
js2 wrote: post #9

2005. So I guess this didn't catch on.

Ubuntu still provides zsync of its installation media: http://cdimage.ubuntu.com/ubuntu/releases/17.04/release/ That said, I'm not sure I know of any other major users of it -- most people just use a .torrent (which similarly has checksums of each piece so you know which pieces need to be downloaded).

> which similarly has checksums of each piece so you know which pieces need to be downloaded

Can you achieve differential downloads with bittorrent?

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#39

Post by cat199 »

rripken wrote: post #36
IanCal wrote: post #16

Earlier quoted context omitted.

Interesting note on the site is that these files don't need to be on the original server. Anyone can generate and host them. > in fact, the .zsync can be generated and offered by a third party, while still leaving most of the download to the original distribution site.

Interesting. How would that work for clients? If the destination file already exists on the client look for a .zsync on the hosting server, if its not there look for one at https://thirdpartyzsyncs.com?url=someurl ? What happens if the .zsync is out of sync with the resource?

my uninformed guess is that the zsync URL would be provided in some HTTP header

Re: Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

#40

Post by iFire »

eeZah7Ux wrote: post #37
iFire wrote: post #32

I'm not related to Itchio. If you are looking for a maintained system for online systems that provide software updates I would look into https://github.com/itchio/wharf-spec . Wharf is used for Itchio to sync folder structures differentially / incrementally. It uses the latest compression algorithms. It has a reference server. Alternatively, a port of zsync is https://github.com/salesforce/zsync4j written in Java. I…

How is the wharf protocol better than zsync?

https://itch.io/docs/wharf/design-goals.html Describes the goals.

https://itch.io/docs/wharf/algorithms/diff.html and https://itch.io/docs/wharf/algorithms/apply.html describe patching

The most important thing is that Itchio runs a business on the usability of this system.

Post reply on HN