Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC File Transfer

github.com

Post reply on HN
Print view View original

51–60 of 106 posts

Re: CDC File Transfer

#51

Post by MayeulC »

I am quite confused; doesn't rsync already use content-defined chunk boundaries, with a condition on the rolling hash to define boundaries?

https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli...

https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli...

The speed improvements over rsync seem related to a more efficient rolling hash algorithm, and possibly by using native windows executables instead of cygwin (windows file systems are notoriously slow, maybe that plays a role here).

Or am I missing something?

In any case, the performance boost is interesting. Glad the source was opened, and I hope it finds its way into rsync.

Re: CDC File Transfer

#52

Post by rokkamokka »

EdSchouten wrote: post #23

I’ve also been doing lots of experimenting with Content Defined Chunking since last year (for https://bonanza.build/ ). One of the things I discovered is that the most commonly used algorithm FastCDC (also used by this project) can be improved significantly by looking ahead. An implementation of that can be found here: https://github.com/buildbarn/go-cdc

What would you estimate the performance implications of using go-cdc instead of fastcdc in their cdc_rsync are?

Re: CDC File Transfer

#53

Post by sneak »

MayeulC wrote: post #51

I am quite confused; doesn't rsync already use content-defined chunk boundaries, with a condition on the rolling hash to define boundaries? https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli... https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli... The speed improvements over rsync seem related to a more efficient rolling hash algorithm, and possibly by using native windows executables instead of c…

rsync seems frozen in time; it’s been around for ages and there are so many basic and small quality of life improvements that could have been made that haven’t been. I have always assumed it’s like vim now: only really maintained in theory, not in practice.

Re: CDC File Transfer

#54

Post by [deleted] »

MayeulC wrote: post #51

I am quite confused; doesn't rsync already use content-defined chunk boundaries, with a condition on the rolling hash to define boundaries? https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli... https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli... The speed improvements over rsync seem related to a more efficient rolling hash algorithm, and possibly by using native windows executables instead of c…

[deleted]

Re: CDC File Transfer

#56

Post by Zardoz84 »

sneak wrote: post #53
MayeulC wrote: post #51

I am quite confused; doesn't rsync already use content-defined chunk boundaries, with a condition on the rolling hash to define boundaries? https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli... https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli... The speed improvements over rsync seem related to a more efficient rolling hash algorithm, and possibly by using native windows executables instead of c…

rsync seems frozen in time; it’s been around for ages and there are so many basic and small quality of life improvements that could have been made that haven’t been. I have always assumed it’s like vim now: only really maintained in theory, not in practice.

So you not used vim or neovim in the last 10 years ?

Re: CDC File Transfer

#57

Post by maccard »

ur-whale wrote: post #22

Great initiative, especially the new sync algorithm, but giant hurdles to adoption: - only works on a weird combo of (src platform / dst platform). Why???? How hard is it to write platform-independent code to read/write bytes and send them over the wire in 2025? - uses bazel, an enormous, Java-based abomination, to build. Fingers crossed that these can be fixed, or this project is dead in the water.

> only works on a weird combo of (src platform / dst platform). Why????

Stadia ran on linux, and 99.9999999% of game development is done on windows (and cross compiled for linux).

> Fingers crossed that these can be fixed, or this project is dead in the water.

The project was archived 9 months ago, and hasn't had a commit in 2 years. It's already dead.

Re: CDC File Transfer

#58

Post by adzm »

oofbey wrote: post #7
theamk wrote: post #3

This CDC is "Content Defined Chunking" - fast incremental file transfer. Use case is to copy file over slow net, but the previous version is already there, so one can save time by only sending changed parts of the file. Not to be confused with USB CDC ("communications device class"), an USB device protocol used to present serial ports and network cards. It can also be used to transfer files, the old PC-to-PC cables u…

The clever trick is how it recognizes insertions. The standard trick of computing hashes on fixed sized blocks works efficiently for substitutions but is totally defeated by an insertion or deletion. Instead with CDC the block boundaries are define by the content, so an insertion doesn’t change the block boundary, so it can tell the subsequent blocks are unchanged. I haven’t read the CDC paper but I’m guessing they j…

> I haven’t read the CDC paper but I’m guessing they just use some probabilistic hash function to define certain strings as block boundaries.

You choose a number of bits (say, 12) and then evenly distribute these in a 48-bit mask; if the hash at any point has all these bits on, that defines a boundary.

Re: CDC File Transfer

#59

Post by bilekas »

This is actually kind of cool, I've implemented my own version of this for my job and seems to be something that's important when the numbers gets tight, but if I remember correctly for their case i guess, wouldn't it have been easier to work from rsynch?

> scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression.

I havent tried it myself but doesnt this already suit that requirement ? https://docs.rc.fas.harvard.edu/kb/rsync/

> Compression If the SOURCE and DESTINATION are on different machines with fast CPUs, especially if they’re on different networks (e.g. your home computer and the FASRC cluster), it’s recommended to add the -z option to compress the data that’s transferred. This will cause more CPU to be used on both ends, but it is usually faster.

Maybe it's not fast enough, but seems a better place to start than scp imo.

Re: CDC File Transfer

#60

Post by Sammi »

It's dead and archived atm, but it looks like a good candidate for revival as an actual active open source project. If you ever wanted to work on something that looks good on your resume, then this looks like your chance. Basically just get it running and released on all major platforms.
Post reply on HN