The rsync algorithm (1996) [pdf] • Hacker News Board

Re: The rsync algorithm (1996) [pdf]

#21

Post by ssl-3 » Sat, Jan 03, 2026, 11:37 AM UTC

The first time I got paid to use rsync was nearly 25 years ago. It provided for reasonably space-efficient, remote, versioned backups of a mail server, using hard links.

That mail server used maildir, which...for those who are not familiar: With maildir, each email message is a separate file on the disk. Thus, there were a lot of folders that had many thousands of files in them. Plus hardlinks for daily/weekly/whatever versions of each of those files.

At the time there were those who were very vocal about their opinion of using maildir in this kind of capacity, likening it to abuse of the filesystem. And if that was stupid, then my use of hard links certainly multiplied that stupidity.

Perhaps I was simply not very smart at that time.

But it was actually fun to fit that together, and it was kind of amazing to watch rsync perform this job both automatically and without complaint between a pair of particularly not-fast (256kbps?) DOCSIS connections from Roadrunner.

It worked fine. Whenever I needed to go back in time for some reason, the information was reliably present at the other end with adequate granularity -- with just a couple of cron jobs, rsync, and maybe a little bit of bash script to automate it all.

Re: The rsync algorithm (1996) [pdf]

#22

Post by webdevver » Sat, Jan 03, 2026, 11:53 AM UTC

teleforce wrote: post #8

Fun facts, the author of rsync, Andrew Tridgell, is also the one who reverse-engineered Microsoft SMB that laid the foundation for Samba [1]. How he did manage to avoid lawsuits from Microsoft is beyond me. [1] Server Message Block: https://en.wikipedia.org/wiki/Server_Message_Block

>How he did manage to avoid lawsuits from Microsoft is beyond me.

MS probably chose not to shut down that effort on the basis that it was enabling the MS stack in Linux.

I wish I could dig up an internal presentation that was prepared in the 90s for Bill Gates at the time, which evaluated the threat posed by Linux to Microsoft. I think they were probably happy that Linux now had a reason to talk to Windows machines.

Re: The rsync algorithm (1996) [pdf]

#23

Post by YesThatTom2 » Sat, Jan 03, 2026, 12:16 PM UTC

teleforce wrote: post #8

Fun facts, the author of rsync, Andrew Tridgell, is also the one who reverse-engineered Microsoft SMB that laid the foundation for Samba [1]. How he did manage to avoid lawsuits from Microsoft is beyond me. [1] Server Message Block: https://en.wikipedia.org/wiki/Server_Message_Block

At first, MS didn’t mind as long as SAMBA only implemented the outdated older protocols.

Then they realized interoperability could make them more money, and they invited him and his team to Redmond for a week of working with MS engineers to understand the latest protocol versions. Oh wait, no, it was because the EU forced them. https://www.theregister.com/2007/12/21/samba_microsoft_agree...

Re: The rsync algorithm (1996) [pdf]

#24

Post by alright2565 » Sat, Jan 03, 2026, 1:32 PM UTC

craftkiller wrote: post #6

I've been using this extensively recently. I was setting up remote virtual machines that boot a live ISO containing all the software for the machine. Sometimes I need to change a small config file, which would lead to generating a new 1.7GiB ISO, but 99.9% of that ISO is identical to the previous one. So I used rsync. Blew my mind when after a day of working on these images, uploading 1.7GiB ISO after 1.7GiB ISO, wir…

If it was a 100GB image on the other hand—good luck! It'd be faster to copy it from scratch every time than to use rsync.

Re: The rsync algorithm (1996) [pdf]

#25

Post by mickael-kerjean » Sat, Jan 03, 2026, 1:37 PM UTC

In section 6: "tar files ... of the Linux kernel sources ... version ... 1.99.10 ... are approximately 24MB in size ... Out of the 2441 files in the 2.0.0 release 291 files had changed"

It never crossed my mind Linux at some point only had 2441 files and you could actually parse the code that went through a new version, that time has sailed

Re: The rsync algorithm (1996) [pdf]

#26

Post by alex1138 » Sat, Jan 03, 2026, 3:17 PM UTC

1996 is not that long ago as Unix goes but it's fun to know as I browse with my Debian computer that I'm using something (derived from Unix but BSD also isn't original Unix either) that has a long tradition

Re: The rsync algorithm (1996) [pdf]

#27

Post by jrpelkonen » Sat, Jan 03, 2026, 3:20 PM UTC

axiolite wrote: post #10

PunchyHamster wrote: post #3

Earlier quoted context omitted.

to be fair level of security of systems back then was pretty fucking bad

In 1996? OpenBSD and Apache had been around for a year. PGP had been around for several years. HTTPS was used where needed. SecurID tokens were common for organizations that cared about security. Admittedly SSH wasn't around, but kerberos+rlogin and SSL+telnet was available. Organizations who cared about security would have SecurID tokens issued to their employees and required for login. Dial-in over phone lines, and…

SSH was around, but not nearly as pervasive it is today. I have memories of having to shake my mouse around during the windows client installation to generate entropy. Fun times

Re: The rsync algorithm (1996) [pdf]

#28

Post by tekkk » Sat, Jan 03, 2026, 3:54 PM UTC

js2 wrote: post #12

teleforce wrote: post #8

Fun facts, the author of rsync, Andrew Tridgell, is also the one who reverse-engineered Microsoft SMB that laid the foundation for Samba [1]. How he did manage to avoid lawsuits from Microsoft is beyond me. [1] Server Message Block: https://en.wikipedia.org/wiki/Server_Message_Block

He also wrote a free BitKeeper client, antagonizing Larry McVoy, which is largely why we have git. https://blog.brachiosoft.com/en/posts/git/

Now that was an awesome blog post, thank you for linking!

Re: The rsync algorithm (1996) [pdf]

#29

Post by amiga386 » Sat, Jan 03, 2026, 4:39 PM UTC

teleforce wrote: post #8

Fun facts, the author of rsync, Andrew Tridgell, is also the one who reverse-engineered Microsoft SMB that laid the foundation for Samba [1]. How he did manage to avoid lawsuits from Microsoft is beyond me. [1] Server Message Block: https://en.wikipedia.org/wiki/Server_Message_Block

He describes how he did it with a French Café analogy:

https://download.samba.org/pub/tridge/misc/french_cafe.txt

Re: The rsync algorithm (1996) [pdf]

#30

Post by adrian_b » Sat, Jan 03, 2026, 5:07 PM UTC

imiric wrote: post #17

Rsync is one of my favorite programs. I use it daily. The CLI is a bit quirky (e.g. trailing slashes), but once you get used to it, it makes sense. And I really always use the same flags: `-avmLP`, with `-n` for dry runs. One alternative I'd like to try is Google's abandoned CDC[1], which claims to be up to 30x faster than rsync in certain scenarios. Does anyone know if there is a maintained fork with full Linux supp…

The same is true for me.

I always alias rsync to:

'/usr/bin/rsync --archive --xattrs --acls --hard-links --progress --rsh="ssh -p PORT -l USER"'

I almost never use any other program for file transfers between computers.