Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does rsync work?

michael.stapelberg.ch

Post reply on HN
Print view View original

41–50 of 54 posts

Re: How does rsync work?

#41

Post by shakna »

londons_explore wrote: post #8
boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

Failure cases of the 'quick check': * Underlying disk device corruption - but modern disks do internal error checking, and should emit an IO error. * Corruption in RAM/software bug in the kernel IO subsystem. Should be detected by filesystem checksumming. * User has accidentally modified file and set mtime back. fixes this case . * User has maliciously modified file and set mtime back. Since it's MD5 (broken), the ma…

Or they're using a filesystem where mtime is always zero, i.e. unsupported.

Which is an actual case that has occured for myself.

Re: How does rsync work?

#42

Post by hinkley »

throw0101a wrote: post #16
boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

> For protocol 30 and beyond (first supported in 3.0.0), the checksum used is MD5. For older protocols, the checksum used is MD4. Newer versions (≥3.2?) support xxHash and xxHash3: * https://github.com/WayneD/rsync/blob/master/checksum.c * https://github.com/Cyan4973/xxHash * https://news.ycombinator.com/item?id=19402602 (2019 XXH3 discussion)

I was surprised to learn recently that sha1 is typically about 10% faster than md5. I have some coworkers who I need to remind of this, since they’re still using md5 for caching.

Re: How does rsync work?

#43

Post by Izkata »

londons_explore wrote: post #8
boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

Failure cases of the 'quick check': * Underlying disk device corruption - but modern disks do internal error checking, and should emit an IO error. * Corruption in RAM/software bug in the kernel IO subsystem. Should be detected by filesystem checksumming. * User has accidentally modified file and set mtime back. fixes this case . * User has maliciously modified file and set mtime back. Since it's MD5 (broken), the ma…

It has its use: I discovered corruption in files copied between external hard drives over a USB hub, using checksum. It was a bad hub, hasn't happened again since I changed it (finding it once got me paranoid enough to check every once in a while).

For anyone curious about how to find such problems without changing the files, I used "--checksum --dry-run --itemize-changes"

Re: How does rsync work?

#44

Post by JonathonW »

throw0101a wrote: post #16
boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

> For protocol 30 and beyond (first supported in 3.0.0), the checksum used is MD5. For older protocols, the checksum used is MD4. Newer versions (≥3.2?) support xxHash and xxHash3: * https://github.com/WayneD/rsync/blob/master/checksum.c * https://github.com/Cyan4973/xxHash * https://news.ycombinator.com/item?id=19402602 (2019 XXH3 discussion)

As mentioned in a sibling comment, linux.die.net's manpage is outdated here (covering rsync 3.0.6). Current versions of rsync (>= 3.2.0) autonegotiate checksum type between several different options (including a few variants of xxhash, md5, and md4) unless overridden by the user, or one side is pre-3.2.0 (in which case you get the old behavior with MD5 or MD4 depending on protocol version). The pre-transfer and transfer checksums also don't necessarily have to be the same (I don't particularly care to hunt down the default priority lists right now, so I'm not sure if they are the same or not by default).

Manpage for rsync 3.2.4, from Debian testing: https://manpages.debian.org/testing/rsync/rsync.1.en.html

Re: How does rsync work?

#45

Post by yencabulator »

hinkley wrote: post #42
throw0101a wrote: post #16

Earlier quoted context omitted.

> For protocol 30 and beyond (first supported in 3.0.0), the checksum used is MD5. For older protocols, the checksum used is MD4. Newer versions (≥3.2?) support xxHash and xxHash3: * https://github.com/WayneD/rsync/blob/master/checksum.c * https://github.com/Cyan4973/xxHash * https://news.ycombinator.com/item?id=19402602 (2019 XXH3 discussion)

I was surprised to learn recently that sha1 is typically about 10% faster than md5. I have some coworkers who I need to remind of this, since they’re still using md5 for caching.

And Blake3 is almost 7 times faster than SHA-1...

Re: How does rsync work?

#46

Post by GekkePrutser »

kzrdude wrote: post #5
boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

I guess zfs send and similar are better solutions, but what if we could query the filesystem for existing checksums of a file and save IO that way, if filesystems on both sides already stored usable checksums?

ZFS send does not do any kind of cross checking with the receiving end so no, not really ideal. Even incremental ZFS sends don't do this, it keeps state on the sending side only. It's ok for its intended usecases but it's not a direct rsync replacement.

It's also hard/impossible to restore individual files out of a ZFS send stream without restoring the whole thing so I've reverted to using tarballs of ZFS snapshots for backups instead of ZFS send. Again, it was never really meant for this so it was my mistake trying to use it that way.

Re: How does rsync work?

#47

Post by GekkePrutser »

boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

Yeah I always do the quick check normally but once in a while I run a full rescan to make sure nothing got damaged over time. Definitely worth it.

Re: How does rsync work?

#48

Post by mrb »

srvmshr wrote: post #13

I encountered a strange situation 2 days ago. I rsync my pdf files periodically between my harddrives. rsync showed no differences between two folder trees, but if I did `diff -r` between the two, 3 pdfs came out different. I checked the three individually but they showed no corruption or changes either side. How can this happen? Edit: the hard drive copy is previously rsynced from this copy & both copies are mirrore…

If your PDF files have the same MD5 checksum but "diff" shows differences then this is an MD5 collision.

Maybe it's a trivial thing, eg. your 3 files got resynchronized right between running rsync and running diff. So you should have retried rsync after the diff.

Or you obtained these PDFs from a source that purports to demonstrate MD5 collisions. Or someone is attacking you by replacing your files. Or, more likely, this is user error, and you are not reporting to us what's happening exactly.

You can always do a diff on a hex dump of the PDF content and see with your own eyes what part of the PDF is actually different. It's not that hard to interpret the format and know which PDF structure changed. You can run "qpdf --qfd input.pdf" on both versions and this uncompress all structures to make the internal content human readable (besides images).

Re: How does rsync work?

#49

Post by srvmshr »

mrb wrote: post #48
srvmshr wrote: post #13

I encountered a strange situation 2 days ago. I rsync my pdf files periodically between my harddrives. rsync showed no differences between two folder trees, but if I did `diff -r` between the two, 3 pdfs came out different. I checked the three individually but they showed no corruption or changes either side. How can this happen? Edit: the hard drive copy is previously rsynced from this copy & both copies are mirrore…

If your PDF files have the same MD5 checksum but "diff" shows differences then this is an MD5 collision. Maybe it's a trivial thing, eg. your 3 files got resynchronized right between running rsync and running diff. So you should have retried rsync after the diff. Or you obtained these PDFs from a source that purports to demonstrate MD5 collisions. Or someone is attacking you by replacing your files. Or, more likely,…

I tried and the difference between the two turns out all gibberish.

>Or, more likely, this is user error, and you are not reporting to us what's happening exactly.

Here's the sequence:

1. rsync -av src/ dest/

2. diff -r src/ dest/

[Shows 3 pairs of differences]

3.

   md5 file1a & file 1b; compare
   md5 file2a & file 2b; compare
   md5 file3a & file 3b; compare
[All three pairs match MD5]

4. run rsync - no difference still

5.compare difference by diff - shows gibberish. All three copies open by pdf viewers. The qpdf option doesn't make sense because all the 3 happen to be advanced math textbooks, and the plaintext is impossible to read.

In fact I just checked now, and the same error pattern persists. Its not something I am terribly concerned since I have 3 copies of my data - but this is a pattern which showed up for the first time. I do this exercise regularly (once per month)

Re: How does rsync work?

#50

Post by jwilk »

mrb wrote: post #48
srvmshr wrote: post #13

I encountered a strange situation 2 days ago. I rsync my pdf files periodically between my harddrives. rsync showed no differences between two folder trees, but if I did `diff -r` between the two, 3 pdfs came out different. I checked the three individually but they showed no corruption or changes either side. How can this happen? Edit: the hard drive copy is previously rsynced from this copy & both copies are mirrore…

If your PDF files have the same MD5 checksum but "diff" shows differences then this is an MD5 collision. Maybe it's a trivial thing, eg. your 3 files got resynchronized right between running rsync and running diff. So you should have retried rsync after the diff. Or you obtained these PDFs from a source that purports to demonstrate MD5 collisions. Or someone is attacking you by replacing your files. Or, more likely,…

Typo: it's "--qdf".
Post reply on HN