CDC File Transfer

Maybe I’m projecting (probably) but I detect some subtext in the README’s language. Like a “developers who are irate that Stadia was shut down, wanting to free anything they can for the community” type feeling.

Re: CDC File Transfer

#5

Post by [deleted] » Sun, Jan 08, 2023, 10:07 PM UTC

[deleted]

Re: CDC File Transfer

#6

Post by jws » Sun, Jan 08, 2023, 10:09 PM UTC

brendoncarroll wrote: post #3

FastCDC is the same chunking algorithm used in Got. https://github.com/gotvc/got

To elaborate, rsync chunks in fixed sizes, so inserting or deleting a few bytes makes all different chunks from that point onward.

If instead you chunk based off of local content (conceptually like chunking text into sentences at periods, but its a binary thing on has an upper size limit and lower size limit and I couldn't find the algorithm specification) so that after an insertion or deletion in a small number of bytes you start getting the same chunks as before.

This drastically reduces the cost of identifying unmodified chunks.

Re: CDC File Transfer

#7

Post by elromulous » Sun, Jan 08, 2023, 10:09 PM UTC

Waterluvian wrote: post #4

Maybe I’m projecting (probably) but I detect some subtext in the README’s language. Like a “developers who are irate that Stadia was shut down, wanting to free anything they can for the community” type feeling.

Google / abc has a solid track record of open sourcing shutdown projects. See Makani[1] and iirc also Loon.

[1] https://news.ycombinator.com/item?id=24456613

Re: CDC File Transfer

#8

Post by progbits » Sun, Jan 08, 2023, 10:11 PM UTC

sigmonsays wrote: post #2

can we gut rsync and implement this? rsync doesn't even have a great protocol specification... maybe it's a new tool...

> Overall, cdc_rsync syncs files about 3 times faster than Cygwin rsync.

Sounds promising.

Re: CDC File Transfer

#9

Post by Waterluvian » Sun, Jan 08, 2023, 10:12 PM UTC

elromulous wrote: post #7

Waterluvian wrote: post #4

Maybe I’m projecting (probably) but I detect some subtext in the README’s language. Like a “developers who are irate that Stadia was shut down, wanting to free anything they can for the community” type feeling.

Google / abc has a solid track record of open sourcing shutdown projects. See Makani[1] and iirc also Loon. [1] https://news.ycombinator.com/item?id=24456613

Aha! I wasn’t at all aware of that. Okay, this reads more like that to me second time around. Just excitement to open source a cool piece of tech that came from Stadia.

Re: CDC File Transfer

#10

Post by teraflop » Sun, Jan 08, 2023, 10:14 PM UTC

TL;DR they rediscovered the context-dependent variable block size technique that Tarsnap uses: https://www.tarsnap.com/download/EuroBSDCon13.pdf