Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The rsync algorithm (1996) [pdf]

andrew.cmu.edu

Post reply on HN
Print view View original

41–50 of 51 posts

Re: The rsync algorithm (1996) [pdf]

#41

Post by jraph »

xk3 wrote: post #35
ssl-3 wrote: post #21

The first time I got paid to use rsync was nearly 25 years ago. It provided for reasonably space-efficient, remote, versioned backups of a mail server, using hard links. That mail server used maildir, which...for those who are not familiar: With maildir, each email message is a separate file on the disk. Thus, there were a lot of folders that had many thousands of files in them. Plus hardlinks for daily/weekly/whatev…

> there were a lot of folders that had many thousands of files in them If you ever need to do something like this again, it's often faster to parallelize rsync. One tool that provides this is fpsync: https://www.fpart.org/fpsync/

And you'd probably use the snapshot feature of a filesystem like btrfs or zfs instead of hardlinks for deduplication :-)

Re: The rsync algorithm (1996) [pdf]

#42

Post by yencabulator »

craftkiller wrote: post #6

I've been using this extensively recently. I was setting up remote virtual machines that boot a live ISO containing all the software for the machine. Sometimes I need to change a small config file, which would lead to generating a new 1.7GiB ISO, but 99.9% of that ISO is identical to the previous one. So I used rsync. Blew my mind when after a day of working on these images, uploading 1.7GiB ISO after 1.7GiB ISO, wir…

> Fun surprise, rsync uses file size and modified time first to see if the files are identical. [...] time to Jan 1st 1970 for reproducible builds

I think it'd be a good idea for rsync to not trust timestamp 0.

Re: The rsync algorithm (1996) [pdf]

#44

Post by itsthecourier »

jbotz wrote: post #39
axiolite wrote: post #33

Earlier quoted context omitted.

I believe your recollection is off by several years... What you're describing is PuttyGen. According to Wikipedia, the first Putty release was in 1999. Archive.org doesn't have any snapshots of the Putty website before 2000, so that checks-out. The RSA patent didn't expire in the US until September 2000, so that's when free implementations like OpenSSH first became widely available. That's precisely when I started us…

> There would have been a small number of installations in 1996, but absolutely negligible. On HN there's always a good chance you're talking to some of the people involved in those "negligible" installations. I know that I submitted some patches to Tatu Ylönen for Ssh to compile on Ultrix, so that must have been in 1995 or early 1996 because after that I didn't have access to any Ultrix machines. I may have been an…

so I was running a SVN server in a decommissioned PC somewhere in a startup as an intern. whole company ends up using it and out of nowhere it used to freeze, I would go to check if it had rebooted or crashed and everything was fine.

it fixed by itself, without any fixes from my part. happened many times.

asked for help to a senior, guy ran strace and found a read waiting in /dev/random. and of course it solved by itself any time I checked because I was moving the mouse!

controversially but acceptably, we had linked it to urandom and move on

how fast that guy used strace and analyzed the syscalls inspired me to be better at linux

Re: The rsync algorithm (1996) [pdf]

#45

Post by webdevver »

amiga386 wrote: post #40
webdevver wrote: post #22

Earlier quoted context omitted.

>How he did manage to avoid lawsuits from Microsoft is beyond me. MS probably chose not to shut down that effort on the basis that it was enabling the MS stack in Linux. I wish I could dig up an internal presentation that was prepared in the 90s for Bill Gates at the time, which evaluated the threat posed by Linux to Microsoft. I think they were probably happy that Linux now had a reason to talk to Windows machines.

https://en.wikipedia.org/wiki/Halloween_documents -> https://www.gnu.org/software/fsfe/projects/ms-vs-eu/hallowee...

thats the one, thankyou for posting!

Re: The rsync algorithm (1996) [pdf]

#46

Post by axiolite »

SoftTalker wrote: post #38
axiolite wrote: post #36

Earlier quoted context omitted.

You'll find something like BorgBackup will be far more efficient than rsync.

But rsync is widely available, usually installed by default on linux or unix-like systems. You can just use it.

Borg is available for download as a standalone binary, easily dropped onto any Linux system even with very limited privs. And in the repos of every distro easily installed and kept up-to-date.

By avoiding that one step and using rsync instead, you're resigning yourself to "send 600MiBs" over the network for every tiny config change. Not a good trade-off.

Re: The rsync algorithm (1996) [pdf]

#47

Post by axiolite »

jbotz wrote: post #39
axiolite wrote: post #33

Earlier quoted context omitted.

I believe your recollection is off by several years... What you're describing is PuttyGen. According to Wikipedia, the first Putty release was in 1999. Archive.org doesn't have any snapshots of the Putty website before 2000, so that checks-out. The RSA patent didn't expire in the US until September 2000, so that's when free implementations like OpenSSH first became widely available. That's precisely when I started us…

> There would have been a small number of installations in 1996, but absolutely negligible. On HN there's always a good chance you're talking to some of the people involved in those "negligible" installations. I know that I submitted some patches to Tatu Ylönen for Ssh to compile on Ultrix, so that must have been in 1995 or early 1996 because after that I didn't have access to any Ultrix machines. I may have been an…

> it didn't take long for ssh to take over the world

That doesn't seem to be accurate. Wikipedia says, by the end of "2000 the number of users had grown to 2 million"

> everybody was using ssh because there wasn't any alternative

I already listed TWO of the most popular alternatives.

> the mouse-jiggling thing... not specifically a PuttyGen thing. On linux

Parent specifically said "windows client installation." Putty was very common on Windows. PuttyGen specifically and prominently told the user to move their mouse... etc. etc.

Re: The rsync algorithm (1996) [pdf]

#48

Post by OrangeDelonge »

linsomniac wrote: post #31

If you want to do similar for block devices: https://github.com/rolffokkens/bdsync I use it to back up a few virtual machines that, in the event of a site loss, would be difficult to rebuild but also critical to getting our developers back to work. I take an LVM snapshot of the VM, then use bdsync to replicate it to our backup server, and from there I replicate it off to backblaze, then destroy the snapshot.

How does this compare to drbd?

Re: The rsync algorithm (1996) [pdf]

#49

Post by ectospheno »

irusensei wrote: post #32
snvzz wrote: post #15

Besides Tridgell's venerable rsync, there exists a permissively licensed implementation[0] by openbsd. 0. https://www.openrsync.org/

Which is also the current version on MacOS. Thing is it doesn't seem to talk very well with the samba version of rsync. The OpenBSD implementation seems to be capped at the version 29 of the protocol. When pulling data on MacOS from a Linux computer I would experiment hangs even when setting protocol versions to 29 or 28. What fixed for me was to just switch to the samba rsync program on MacOS.

I believe openrsync exists just because of rpki.

https://news.ycombinator.com/item?id=43605846

Most openbsd people I know install the real version from ports.

Re: The rsync algorithm (1996) [pdf]

#50

Post by linsomniac »

OrangeDelonge wrote: post #48
linsomniac wrote: post #31

If you want to do similar for block devices: https://github.com/rolffokkens/bdsync I use it to back up a few virtual machines that, in the event of a site loss, would be difficult to rebuild but also critical to getting our developers back to work. I take an LVM snapshot of the VM, then use bdsync to replicate it to our backup server, and from there I replicate it off to backblaze, then destroy the snapshot.

How does this compare to drbd?

DRBD is more of a live sync, and it's great stuff, as long as you set it up BEFORE you need it, and you need it frequently. If you want to keep a second copy of your data on another system, up to the second(ish), it's a great choice.

If, however, you just want a copy of a block device on another system, like for weekly backup (our case), it's probably overkill. Especially as to keep it truly consistent you need to run in the mode where writes are acked only once the remote AND local devices have it.

My VMs are running on ganeti, which has a mode where the backing device can be DRBD and written to another host. Which works great if you have the extra disc space and can deal with the latency. Also allows you to live migrate VMs between the two hosts.

In my case I ultimately want the copy off-site, so DRBD isn't really a great fit.

DRBD is very good stuff though, I've used it for decades for HA database servers and the like.

Post reply on HN