Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does rsync work?

michael.stapelberg.ch

Post reply on HN
Print view View original

11–20 of 54 posts

Re: How does rsync work?

#11

Post by js2 »

kzrdude wrote: post #5
boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

I guess zfs send and similar are better solutions, but what if we could query the filesystem for existing checksums of a file and save IO that way, if filesystems on both sides already stored usable checksums?

> I guess zfs send and similar are better solutions.

It depends. I recently built a new zfs pool server and needed to transfer a few TB of data from the old pool to the new pool, but I built the new pool with a larger record size. If I’d used zfs send the files would have retained their existing record size. So rsync it was.

Re: How does rsync work?

#12

Post by lloeki »

When trying to understand rsync and the rolling checksum I stumbled upon a small python implementation in some self-hosted corner of the web[0], which I have archived on GH[1] (not the author, but things can vanish quickly, as proved by the bzr repo which went poof[2]).

[0]: https://blog.liw.fi/posts/rsync-in-python/

[1]: https://github.com/lloeki/rsync/blob/master/rsync.py

[2]: https://code.liw.fi/obsync/bzr/trunk/

Re: How does rsync work?

#13

Post by srvmshr »

I encountered a strange situation 2 days ago. I rsync my pdf files periodically between my harddrives. rsync showed no differences between two folder trees, but if I did `diff -r` between the two, 3 pdfs came out different.

I checked the three individually but they showed no corruption or changes either side. How can this happen?

Edit: the hard drive copy is previously rsynced from this copy & both copies are mirrored with google cloud bucket.

The 3 files which showed different have the same MD5 checksum

Re: How does rsync work?

#14

Post by orangepurple »

srvmshr wrote: post #13

I encountered a strange situation 2 days ago. I rsync my pdf files periodically between my harddrives. rsync showed no differences between two folder trees, but if I did `diff -r` between the two, 3 pdfs came out different. I checked the three individually but they showed no corruption or changes either side. How can this happen? Edit: the hard drive copy is previously rsynced from this copy & both copies are mirrore…

rsync uses a heuristic based on file times and sizes to compare files. to compare file content use the --checksum feature (computationally expensive to run)

Re: How does rsync work?

#15

Post by mmlb »

lloeki wrote: post #12

When trying to understand rsync and the rolling checksum I stumbled upon a small python implementation in some self-hosted corner of the web[0], which I have archived on GH[1] (not the author, but things can vanish quickly, as proved by the bzr repo which went poof [2]). [0]: https://blog.liw.fi/posts/rsync-in-python/ [1]: https://github.com/lloeki/rsync/blob/master/rsync.py [2]: https://code.liw.fi/obsync/bzr/trunk/

For bzr did you try archive.org? https://web.archive.org/web/20150321194412/https://code.liw....

Re: How does rsync work?

#16

Post by throw0101a »

boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

> For protocol 30 and beyond (first supported in 3.0.0), the checksum used is MD5. For older protocols, the checksum used is MD4.

Newer versions (≥3.2?) support xxHash and xxHash3:

* https://github.com/WayneD/rsync/blob/master/checksum.c

* https://github.com/Cyan4973/xxHash

* https://news.ycombinator.com/item?id=19402602 (2019 XXH3 discussion)

Re: How does rsync work?

#17

Post by jwilk »

boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

> https://linux.die.net/man/1/rsync

linux.die.net is horribly outdated. This particular page is from 2009.

Up-to-date docs are here:

https://download.samba.org/pub/rsync/rsync.1

Re: How does rsync work?

#18

Post by secure »

jwilk wrote: post #17
boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

> https://linux.die.net/man/1/rsync linux.die.net is horribly outdated. This particular page is from 2009. Up-to-date docs are here: https://download.samba.org/pub/rsync/rsync.1

Which version is the samba one? Latest release? Git?

If you want to see the man page of the version in Debian, that would be https://manpages.debian.org/testing/rsync/rsync.1.en.html

Disclaimer: I wrote the software behind manpages.debian.org :)

Re: How does rsync work?

#20

Post by hatware »

bigChris wrote: post #19

Rsync worst issue is someone port scanning and brute force their way into your system. Turn off your port.

Don't most consumer routers have all ports blocked? Who is connecting a computer directly to the modem these days?
Post reply on HN