Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does rsync work?

michael.stapelberg.ch

Post reply on HN
Print view View original

21–30 of 54 posts

Re: How does rsync work?

#21

Post by jwilk »

mmlb wrote: post #15
lloeki wrote: post #12

When trying to understand rsync and the rolling checksum I stumbled upon a small python implementation in some self-hosted corner of the web[0], which I have archived on GH[1] (not the author, but things can vanish quickly, as proved by the bzr repo which went poof [2]). [0]: https://blog.liw.fi/posts/rsync-in-python/ [1]: https://github.com/lloeki/rsync/blob/master/rsync.py [2]: https://code.liw.fi/obsync/bzr/trunk/

For bzr did you try archive.org? https://web.archive.org/web/20150321194412/https://code.liw....

I don't think you can clone that.

Re: How does rsync work?

#22

Post by jwilk »

secure wrote: post #18
jwilk wrote: post #17

Earlier quoted context omitted.

> https://linux.die.net/man/1/rsync linux.die.net is horribly outdated. This particular page is from 2009. Up-to-date docs are here: https://download.samba.org/pub/rsync/rsync.1

Which version is the samba one? Latest release? Git? If you want to see the man page of the version in Debian, that would be https://manpages.debian.org/testing/rsync/rsync.1.en.html Disclaimer: I wrote the software behind manpages.debian.org :)

"This manpage is current for version 3.2.5dev of rsync" – so I guess it's from git.

Re: How does rsync work?

#23

Post by Dylan16807 »

orangepurple wrote: post #14
srvmshr wrote: post #13

I encountered a strange situation 2 days ago. I rsync my pdf files periodically between my harddrives. rsync showed no differences between two folder trees, but if I did `diff -r` between the two, 3 pdfs came out different. I checked the three individually but they showed no corruption or changes either side. How can this happen? Edit: the hard drive copy is previously rsynced from this copy & both copies are mirrore…

rsync uses a heuristic based on file times and sizes to compare files. to compare file content use the --checksum feature (computationally expensive to run)

Rsync can checksum a lot of megabytes per second. In general I'd say the disk IO is much more expensive than the computation.

Re: How does rsync work?

#24

Post by generalizations »

mmlb wrote: post #15
lloeki wrote: post #12

When trying to understand rsync and the rolling checksum I stumbled upon a small python implementation in some self-hosted corner of the web[0], which I have archived on GH[1] (not the author, but things can vanish quickly, as proved by the bzr repo which went poof [2]). [0]: https://blog.liw.fi/posts/rsync-in-python/ [1]: https://github.com/lloeki/rsync/blob/master/rsync.py [2]: https://code.liw.fi/obsync/bzr/trunk/

For bzr did you try archive.org? https://web.archive.org/web/20150321194412/https://code.liw....

It's not actually there.

https://web.archive.org/web/20150321212547/http://code.liw.f...

Re: How does rsync work?

#25

Post by srvmshr »

orangepurple wrote: post #14
srvmshr wrote: post #13

I encountered a strange situation 2 days ago. I rsync my pdf files periodically between my harddrives. rsync showed no differences between two folder trees, but if I did `diff -r` between the two, 3 pdfs came out different. I checked the three individually but they showed no corruption or changes either side. How can this happen? Edit: the hard drive copy is previously rsynced from this copy & both copies are mirrore…

rsync uses a heuristic based on file times and sizes to compare files. to compare file content use the --checksum feature (computationally expensive to run)

Yes but it doesn't answer why rsync & checksum pass the set of files as same, but diff reports them different.

Re: How does rsync work?

#26

Post by tjoff »

londons_explore wrote: post #8
boomskats wrote: post #2

This was a great write up. I've already sent it to a few people. On the question of what happens if a file's contents change after the initial checksum, the man page for rsync[0] has an interesting explanation of the *--checksum* option: > This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a "quick check" that (by default) checks if each fil…

Failure cases of the 'quick check': * Underlying disk device corruption - but modern disks do internal error checking, and should emit an IO error. * Corruption in RAM/software bug in the kernel IO subsystem. Should be detected by filesystem checksumming. * User has accidentally modified file and set mtime back. fixes this case . * User has maliciously modified file and set mtime back. Since it's MD5 (broken), the ma…

The very existence of filesystem checksumming is because your first point isn't always true.

Also, filesystem checksumming does not guard against ram/kernel-bugs. On top of that file system checksumming is very rare.

Re: How does rsync work?

#27

Post by lazypenguin »

Nice write up. rsync is great as an application but I found it more cumbersome to use when wanting to integrate it into my own application. There's librsync but the documentation is threadbare and it requires an rsync server to run. I found bita/bitar (https://github.com/oll3/bita) which is inspired by rsync & family. It works more like zsync which leverages HTTP Range requests so it doesn't require anything running on the server to get chunks. Works like a treat using s3/b2 storage to serve files and get incremental differential updates on the client side!

Re: How does rsync work?

#29

Post by [deleted] »

throw0101a wrote: post #9
formerly_proven wrote: post #6

Earlier quoted context omitted.

Unless you are also doing FS-level deduplication using the same checksums, it generally makes no sense for these to be cryptographic hashes, so they're not necessarily suitable for this purpose. IIRC neither ZFS nor btrfs use cryptographic hashes for checksumming by default.

> on is a short hand for fletcher4 for non-deduped datasets and sha256 for deduped datasets * https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Chec... * https://people.freebsd.org/~asomers/fletcher.pdf * https://en.wikipedia.org/wiki/Fletcher%27s_checksum Strangely enough SHA-512 is actually (50%) faster than -256: > ZFS actually uses a special version of SHA512 called SHA512t256, it uses a different initial v…

[deleted]

Re: How does rsync work?

#30

Post by samoppy »

srvmshr wrote: post #25
orangepurple wrote: post #14

Earlier quoted context omitted.

rsync uses a heuristic based on file times and sizes to compare files. to compare file content use the --checksum feature (computationally expensive to run)

Yes but it doesn't answer why rsync & checksum pass the set of files as same, but diff reports them different.

No post body was provided.
Post reply on HN