How does rsync work? • Hacker News Board

Re: How does rsync work?

#51

Post by pronoiac » Mon, Jul 04, 2022, 7:43 AM UTC

srvmshr wrote: post #49

mrb wrote: post #48

Earlier quoted context omitted.

If your PDF files have the same MD5 checksum but "diff" shows differences then this is an MD5 collision. Maybe it's a trivial thing, eg. your 3 files got resynchronized right between running rsync and running diff. So you should have retried rsync after the diff. Or you obtained these PDFs from a source that purports to demonstrate MD5 collisions. Or someone is attacking you by replacing your files. Or, more likely,…

I tried and the difference between the two turns out all gibberish. >Or, more likely, this is user error, and you are not reporting to us what's happening exactly. Here's the sequence: 1. rsync -av src/ dest/ 2. diff -r src/ dest/ [Shows 3 pairs of differences] 3. md5 file1a & file 1b; compare md5 file2a & file 2b; compare md5 file3a & file 3b; compare [All three pairs match MD5] 4. run rsync - no difference still 5.…

As noted in a sibling comment, you should check out the --checkout / -c option. Specifically, replace:

1. rsync -acv src/ dest/

It will be much slower - using lots of cpu and disk access - but more thorough.

Re: How does rsync work?

#52

Post by mmlb » Mon, Jul 04, 2022, 5:01 PM UTC

generalizations wrote: post #24

mmlb wrote: post #15

Earlier quoted context omitted.

For bzr did you try archive.org? https://web.archive.org/web/20150321194412/https://code.liw....

It's not actually there. https://web.archive.org/web/20150321212547/http://code.liw.f...

Ah I see now, I didn't traverse deep enough whem checking.

Re: How does rsync work?

#53

Post by mmlb » Mon, Jul 04, 2022, 5:02 PM UTC

jwilk wrote: post #21

mmlb wrote: post #15

Earlier quoted context omitted.

For bzr did you try archive.org? https://web.archive.org/web/20150321194412/https://code.liw....

I don't think you can clone that.

True, but I imagined fetching every file with wget or equivalent and then just using it locally. Doesn't work anyway because the data is not all there.

Re: How does rsync work?

#54

Post by mrb » Tue, Jul 05, 2022, 8:44 PM UTC

srvmshr wrote: post #49

mrb wrote: post #48

Earlier quoted context omitted.

If your PDF files have the same MD5 checksum but "diff" shows differences then this is an MD5 collision. Maybe it's a trivial thing, eg. your 3 files got resynchronized right between running rsync and running diff. So you should have retried rsync after the diff. Or you obtained these PDFs from a source that purports to demonstrate MD5 collisions. Or someone is attacking you by replacing your files. Or, more likely,…

I tried and the difference between the two turns out all gibberish. >Or, more likely, this is user error, and you are not reporting to us what's happening exactly. Here's the sequence: 1. rsync -av src/ dest/ 2. diff -r src/ dest/ [Shows 3 pairs of differences] 3. md5 file1a & file 1b; compare md5 file2a & file 2b; compare md5 file3a & file 3b; compare [All three pairs match MD5] 4. run rsync - no difference still 5.…

Would you mind emailing me file1a and file1b assuming you are willing to share them? Send to: m (at) zorinaq.con

It's possible to have a scenario where your synchronization process (involving Google Cloud?) somehow changed the modification time of your files so that both copies in src/ and dst/ have the same timestamp, in that case rsync will not notice they are different (if they also have the same size). Like the other reply said you have to use rsync --checksum or -c to force rsync to compare the content of files.