Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC File Transfer

github.com

Post reply on HN
Print view View original

61–70 of 106 posts

Re: CDC File Transfer

#61

Post by re »

MayeulC wrote: post #51

I am quite confused; doesn't rsync already use content-defined chunk boundaries, with a condition on the rolling hash to define boundaries? https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli... https://en.wikipedia.org/wiki/Rolling_hash#Content-based_sli... The speed improvements over rsync seem related to a more efficient rolling hash algorithm, and possibly by using native windows executables instead of c…

> doesn't rsync already use content-defined chunk boundaries, with a condition on the rolling hash to define boundaries?

No, it operates on fixed size blocks over the destination file. However, by using a rolling hash, it can detect those blocks at any offset within the source file to avoid re-transferring them.

https://rsync.samba.org/tech_report/node2.html

Re: CDC File Transfer

#62

Post by regularfry »

bilekas wrote: post #59

This is actually kind of cool, I've implemented my own version of this for my job and seems to be something that's important when the numbers gets tight, but if I remember correctly for their case i guess, wouldn't it have been easier to work from rsynch? > scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression. I…

> The remote diffing algorithm is based on CDC. In our tests, it is up to 30x faster than the one used in rsync (1500 MB/s vs 50 MB/s).

Re: CDC File Transfer

#63

Post by EdSchouten »

rokkamokka wrote: post #52
EdSchouten wrote: post #23

I’ve also been doing lots of experimenting with Content Defined Chunking since last year (for https://bonanza.build/ ). One of the things I discovered is that the most commonly used algorithm FastCDC (also used by this project) can be improved significantly by looking ahead. An implementation of that can be found here: https://github.com/buildbarn/go-cdc

What would you estimate the performance implications of using go-cdc instead of fastcdc in their cdc_rsync are?

In my case I observed a ~2% reduction in data storage when attempting to store and deduplicate various versions of the Linux kernel source tree (see link above). But that also includes the space needed to store the original version.

If we take that out of the equation and only measure the size of the additional chunks being transferred, it's a reduction of about 3.4%. So it's not an order of magnitude difference, but not bad for a relatively small change.

Re: CDC File Transfer

#64

Post by killingtime74 »

oofbey wrote: post #5
rekttrader wrote: post #2

Nice to see Stadia had some long term benefit. It’s a shame they don’t make a self hosted version but if you did that it’s just piracy in today’s drm world.

What do you mean piracy in the a DRM world. Like being able to share your own PC games through the cloud?

You can share the games you authored all you like. If you bought a license to play them that's another story.

Re: CDC File Transfer

#65

Post by rincebrain »

bilekas wrote: post #59

This is actually kind of cool, I've implemented my own version of this for my job and seems to be something that's important when the numbers gets tight, but if I remember correctly for their case i guess, wouldn't it have been easier to work from rsynch? > scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression. I…

rsync in my experience is not optimized for a number of use cases.

Game development, in particular, often involves truly enormous sizes and numbers of assets, particularly for dev build iteration, where you're sometimes working with placeholder or unoptimized assets, and debug symbol bloated things, and in my experience, rsync scales poorly for speed of copying large numbers of things. (In the past, I've used naive wrapper scripts with pregenerated lists of the files on one side and GNU parallel to partition the list into subsets and hand those to N different rsync jobs, and then run a sync pass at the end to cleanup any deletions.)

Just last week, I was trying to figure out a more effective way to scale copying a directory tree that was ~250k files varying in size between 128b and 100M, spread out across a complicatedly nested directory structure of 500k directories, because rsync would serialize badly around the cost of creating files and directories. After a few rounds of trying to do many-way rsync partitions, I finally just gave the directory to syncthing and let its pregenerated index and watching handle it.

Re: CDC File Transfer

#66

Post by jcul »

wheybags wrote: post #39

If anyone else was left wondering about the details of how CDC actually generates chunks, I found these two blog posts explained the idea pretty clearly: https://joshleeb.com/posts/content-defined-chunking.html https://joshleeb.com/posts/gear-hashing.html

Thanks, I was puzzled by that. They kind of gloss over it in the original link.

Looking forward to reading those.

Re: CDC File Transfer

#69

Post by heavyset_go »

_cenw wrote: post #38
nolok wrote: post #31

Earlier quoted context omitted.

For self hosted remote streaming of game look at Moonlight / Sunshine (Apollo) Stadia required special version of games, so it wouldn't be that useful

It's a shame that virtual / headless displays are such a mess on both Linux and Windows. I use a 32:9 ultrawide and stream to 16:9/16:10 devices, and even with hours of messing around with an HDMI dummy and kscreen-doctor[1] it was still an unreliable mess. Sometimes it wouldn't work when the machine was locked, and sometimes Sunshine wouldn't restore the resolution on the physical monitor (and there's no session tim…

On Linux with an AMD i/dGPU, you can set the `virtual_display` module parameter for `amdgpu`[1] and do what you want without the need for an HDMI dummy or weird software. It's also hardware accelerated.

> virtual_display (charp)

> Set to enable virtual display feature. This feature provides a virtual display hardware on headless boards or in virtualized environments. It will be set like xxxx:xx:xx.x,x;xxxx:xx:xx.x,x. It’s the pci address of the device, plus the number of crtcs to expose. E.g., 0000:26:00.0,4 would enable 4 virtual crtcs on the pci device at 26:00.0. The default is NULL.

[1]https://www.kernel.org/doc/html/latest/gpu/amdgpu/module-par...

Re: CDC File Transfer

#70

Post by quotemstr »

EdSchouten wrote: post #23

I’ve also been doing lots of experimenting with Content Defined Chunking since last year (for https://bonanza.build/ ). One of the things I discovered is that the most commonly used algorithm FastCDC (also used by this project) can be improved significantly by looking ahead. An implementation of that can be found here: https://github.com/buildbarn/go-cdc

I wonder whether there's a role for AI here.

(Please don't hurt me.)

AI turns out to be useful for data compression (https://statusneo.com/creating-lossless-compression-algorith...) and RF modulation optimization (https://www.arxiv.org/abs/2509.04805).

Maybe it'd be useful to train a small model (probably of the SSM variety) to find optimal chunking boundaries.

Post reply on HN