I ran into some of those issues with the chunk size and hash misses when writing bitsync [1], but at the time I didn't want to get too clever with it because I was focused on rsync algorithm compatibility.
This is a cool idea!
CDC File Transfer
11–20 of 106 posts
Re: CDC File Transfer
#12This CDC is "Content Defined Chunking" - fast incremental file transfer. Use case is to copy file over slow net, but the previous version is already there, so one can save time by only sending changed parts of the file. Not to be confused with USB CDC ("communications device class"), an USB device protocol used to present serial ports and network cards. It can also be used to transfer files, the old PC-to-PC cables u…
not to be confused with Center of Disease Control
Re: CDC File Transfer
#13Does Steam do something like this for game updates?
Re: CDC File Transfer
#14Does Steam do something like this for game updates?
Steam unfortunately doesn't use a rolling hash like this (fastcdc, buzhash, etc.), but rather slices files into 1MB chunks, hashes them, and updates at that granularity.
https://partner.steamgames.com/doc/sdk/uploading#AppStructur...
Re: CDC File Transfer
#15Cygwin? Does anyone still use that?
Re: CDC File Transfer
#16> cdc_rsync is a tool to sync files from a Windows machine to a Linux device, similar to the standard Linux rsync.
Does this work Linux to Linux too?
Re: CDC File Transfer
#17This CDC is "Content Defined Chunking" - fast incremental file transfer. Use case is to copy file over slow net, but the previous version is already there, so one can save time by only sending changed parts of the file. Not to be confused with USB CDC ("communications device class"), an USB device protocol used to present serial ports and network cards. It can also be used to transfer files, the old PC-to-PC cables u…
not to be confused with Center of Disease Control
…or cDc[0]
Re: CDC File Transfer
#18Having dabbled in trying to make a quick delta patch system like Steam's, which required me to understand delta patching methods and made small patches to big files in a 10gb+ installation in a few seconds, this is sure is quite interesting!
I wonder if Steam ever decides to supercharge their content handling with some user-space filesystem stuff. With fast connections, there isn't really a reason they couldn't launch games in seconds, streaming data on-demand with smart pre-caching steering based on automatically trained access pattern data. And especially with finely tuned delta patching like this, online game pauses for patching could be almost entirely eliminated. Stop & go instead of a pit stop.
Re: CDC File Transfer
#19> cdc_rsync is a tool to sync files from a Windows machine to a Linux device, similar to the standard Linux rsync. Does this work Linux to Linux too?
Re: CDC File Transfer
#20Nice to see Stadia had some long term benefit. It’s a shame they don’t make a self hosted version but if you did that it’s just piracy in today’s drm world.
Probably wouldn’t have been feasible - I heard developers had to compile their games with Stadia support. Maybe it was an entirely different platform, with its own alternative to DirectX, or maybe had some kind of lightweight emulation (such as Proton) but I remember vaguely the few games I played had custom stadia key bindings (with stadia symbols). They would display like that within the game. So definitely some customization did happen.
This is unlike the model that PlayStation, Xbox and even Nvidia are following - I don’t know about Amazon Luna.