How does rsync work?
michael.stapelberg.ch
How does rsync work?
1–10 of 54 posts
Re: How does rsync work?
#2This was a great write up. I've already sent it to a few people.
On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option:
> This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each file's size and time of last modification match between the sender and receiver. This option changes this to compare a 128-bit checksum for each file that has a matching size. Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the transfer (and this is prior to any reading that will be done to transfer changed files), so this can slow things down significantly.
> The sending side generates its checksums while it is doing the file-system scan that builds the list of the available files. The receiver generates its checksums when it is scanning for changed files, and will checksum any file that has the same size as the corresponding sender's file: files with either a changed size or a changed checksum are selected for transfer.
> Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer "Does this file need to be updated?" check. For protocol 30 and beyond (first supported in 3.0.0), the checksum used is MD5. For older protocols, the checksum used is MD4.
Re: How does rsync work?
#3See also the 1996 original paper by Tridgell (also of Samba fame) and Mackerras:
* https://rsync.samba.org/tech_report/
* https://www.andrew.cmu.edu/course/15-749/READINGS/required/c...
Re: How does rsync work?
#4See also the 1996 original paper by Tridgell (also of Samba fame) and Mackerras: * https://rsync.samba.org/tech_report/ * https://www.andrew.cmu.edu/course/15-749/READINGS/required/c...
Another great resource from Tridgell himself is this Ottawa Linux Symposium talk: http://olstrans.sourceforge.net/release/OLS2000-rsync/OLS200...
Re: How does rsync work?
#5This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…
I guess zfs send and similar are better solutions, but what if we could query the filesystem for existing checksums of a file and save IO that way, if filesystems on both sides already stored usable checksums?
Re: How does rsync work?
#6This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…
I guess zfs send and similar are better solutions, but what if we could query the filesystem for existing checksums of a file and save IO that way, if filesystems on both sides already stored usable checksums?
Unless you are also doing FS-level deduplication using the same checksums, it generally makes no sense for these to be cryptographic hashes, so they're not necessarily suitable for this purpose.
IIRC neither ZFS nor btrfs use cryptographic hashes for checksumming by default.
Re: How does rsync work?
#7This is also available as a video, "Why I wrote my own rsync":
Re: How does rsync work?
#8This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…
Failure cases of the 'quick check':
* Underlying disk device corruption - but modern disks do internal error checking, and should emit an IO error.
* Corruption in RAM/software bug in the kernel IO subsystem. Should be detected by filesystem checksumming.
* User has accidentally modified file and set mtime back. fixes this case.
* User has maliciously modified file and set mtime back. Since it's MD5 (broken), the malicious user can make the checksum match too. checksumming doesn't help.
Given that, I see no users who really benefit from checksumming. It isn't sufficient for anyone with really high data integrity requirements, while also being overkill for typical usecases.
Re: How does rsync work?
#9Earlier quoted context omitted.
I guess zfs send and similar are better solutions, but what if we could query the filesystem for existing checksums of a file and save IO that way, if filesystems on both sides already stored usable checksums?
Unless you are also doing FS-level deduplication using the same checksums, it generally makes no sense for these to be cryptographic hashes, so they're not necessarily suitable for this purpose. IIRC neither ZFS nor btrfs use cryptographic hashes for checksumming by default.
> on is a short hand for fletcher4 for non-deduped datasets and sha256 for deduped datasets
* https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Chec...
* https://people.freebsd.org/~asomers/fletcher.pdf
* https://en.wikipedia.org/wiki/Fletcher%27s_checksum
Strangely enough SHA-512 is actually (50%) faster than -256:
> ZFS actually uses a special version of SHA512 called SHA512t256, it uses a different initial value, and truncates the results to 256 bits (that is all the room there is in the block pointer). The advantage is only that it is faster on 64 bit CPUs.
Re: How does rsync work?
#10This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…
Failure cases of the 'quick check': * Underlying disk device corruption - but modern disks do internal error checking, and should emit an IO error. * Corruption in RAM/software bug in the kernel IO subsystem. Should be detected by filesystem checksumming. * User has accidentally modified file and set mtime back. fixes this case . * User has maliciously modified file and set mtime back. Since it's MD5 (broken), the ma…
> * User has maliciously modified file and set mtime back. Since it's MD5 (broken), the malicious user can make the checksum match too. checksumming doesn't help.
No, md5 is not broken like that (yet). There is no nkown second-preimage attack against md5; the practical collision vulns only affect cases where an attack controls the file content both before and after update.