Any idea how it compares to aspera?
I may give benchmarking a shot later when I get to my computer, but if anyone has any intuition that could be helpful.
CDC File Transfer
61–70 of 229 posts
Re: CDC File Transfer
#62I'm really impressed with the readme file in this repository. Its a master class in effective documentation. Just look at the structure: - 1-2 sentence summary of the project - History (why we built this and didn't just use existing tools) - What we've built, including what it does and how it compares to other tools (rsync in this case). - How to install and use it And the whole thing is full of images and animations…
It’s always amazing to me how little emphasis people give to INTENT on their repo.
Re: CDC File Transfer
#63I'm really impressed with the readme file in this repository. Its a master class in effective documentation. Just look at the structure: - 1-2 sentence summary of the project - History (why we built this and didn't just use existing tools) - What we've built, including what it does and how it compares to other tools (rsync in this case). - How to install and use it And the whole thing is full of images and animations…
[deleted]
Re: CDC File Transfer
#64This looks like a very useful tool with a wide range of applications. Is it Windows to Linux only? It would be so nice if it was system agnostic.
Re: CDC File Transfer
#65[deleted]
Re: CDC File Transfer
#66Earlier quoted context omitted.
To elaborate, rsync chunks in fixed sizes, so inserting or deleting a few bytes makes all different chunks from that point onward. If instead you chunk based off of local content (conceptually like chunking text into sentences at periods, but its a binary thing on has an upper size limit and lower size limit and I couldn't find the algorithm specification) so that after an insertion or deletion in a small number of b…
The gotcha of "inserting or deleting a few bytes" is not in detection, it's in replicating this discovery to the target copy. Say, we have 1GB file and we detected an extra byte at the head of our local copy. Great, what next? We can't replicate this on the receiving end without recopying the file, which is exactly what happens - rsync recreates target file from pieces of its old copy and differences received from th…
Still, with modern NVMe SSD speeds, usually the network will be the bottleneck. My system with a budget WD Blue gets a decent 1800MB/s sequential write (which somehow caused Win10 to freeze and my taskbar to disappear for a few seconds :/ ) and 2600MB/s sequential read, so even if everything else is unoptimized and the file has to be copied it will still take <1s for your hypothetical 1GB file. Copying the file over a 1Gb network link will take an order of magnitude longer.
Re: CDC File Transfer
#67Comparing it with rsync running on Cygwin is a bit unfair, as Cygwin is known to be terribly inefficient. I don't doubt that their CDC based algorithm is faster, but probably not by the margin they claim if Cygwin is taken out of the equation
They addressed that specifically in the README. They tested on Linux and it was worse but had a worse CPU so assumed it was comparable.
Re: CDC File Transfer
#68I'm currently working on a required CDC (Center for Disease Control) reporting function for a COVID test. For a second I thought this article was going to be extremely helpful.
A new CDC (Change Data Capture) approach open-sourced by Google? Quickly I, a subsequently disappointed data engineer, started reading
I similarly assumed something was coming out from BigQuery
Re: CDC File Transfer
#69I'm really impressed with the readme file in this repository. Its a master class in effective documentation. Just look at the structure: - 1-2 sentence summary of the project - History (why we built this and didn't just use existing tools) - What we've built, including what it does and how it compares to other tools (rsync in this case). - How to install and use it And the whole thing is full of images and animations…
This is how I strive to document things. Focus primarily on what the tool is used for and why, not trying to define what it is.
Re: CDC File Transfer
#70> Born from the ashes of Stadia
Love it. Those were some very expensive ashes. I hope more comes from them too.