Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC File Transfer

github.com

Post reply on HN
Print view View original

171–180 of 229 posts

Re: CDC File Transfer

#171

Post by ljusten »

dark-star wrote: post #54

Comparing it with rsync running on Cygwin is a bit unfair, as Cygwin is known to be terribly inefficient. I don't doubt that their CDC based algorithm is faster, but probably not by the margin they claim if Cygwin is taken out of the equation

IIUC, rsync computes a relatively expensive Rabin-Karp rolling hash (https://librsync.github.io/rabinkarp_8c_source.html) and performs a hash map lookup for every byte. Hash map lookups might not be very cache friendly for larger data sets. In comparison, cdc_rsync only computes

  hash = (hash
per byte. That's only a few ops and very cache friendly. The random table only has 256 entries, 8 bytes each, so it easily fits into L1.

Re: CDC File Transfer

#172

Post by ljusten »

dark-star wrote: post #54

Comparing it with rsync running on Cygwin is a bit unfair, as Cygwin is known to be terribly inefficient. I don't doubt that their CDC based algorithm is faster, but probably not by the margin they claim if Cygwin is taken out of the equation

I should also note that we used a fairly fast 100 MB/sec connection to upload the data, so the rsync diffing algorithm running at 50 MB/sec is actually a bottleneck. The difference would be smaller on a slower connection, where the network overhead would dominate the results.

Re: CDC File Transfer

#173

Post by ljusten »

AnonC wrote: post #87

I skimmed through the readme, which explains the concepts quite well, but am unclear on what needs to be installed on each machine (assuming Windows as the source and Linux as the destination). There’s a mention of copying the Linux build output, cdc_rsync_server, to the Windows machine. Why is this needed? And is there something on the Linux machine that needs to be (newly) added in the PATH?

Just uncompress the binaries on the Windows machine and run cdc_rsync. The Linux component, cdc_rsync_server, is deployed automatically on first run. It is scp'ed to ~/.cache/cdc-file-transfer/bin. So nothing has to be installed on the Linux machine.

Re: CDC File Transfer

#174

Post by ljusten »

RyanShook wrote: post #64

This looks like a very useful tool with a wide range of applications. Is it Windows to Linux only? It would be so nice if it was system agnostic.

We're currently adding support for Windows to Windows to cdc_rsync. If there is demand, Linux to Linux would also be possible.

Re: CDC File Transfer

#175

Post by ljusten »

skanga wrote: post #52
Moissanite wrote: post #16

Earlier quoted context omitted.

I thought the same; seems to be perfectly fine as a Linux-to-Linux tool.

Thanks! Does it work Windows to Windows also?

Windows to Windows is being worked on, see https://github.com/google/cdc-file-transfer/compare/main...s....

Linux to Linux is also an option if there is demand, but currently it's Windows to Linux only.

Re: CDC File Transfer

#176

Post by ljusten »

deanCommie wrote: post #125

> At Stadia, game developers had access to Linux cloud instances to run games. Most developers wrote their games on Windows, though. Therefore, they needed a way to make them available on the remote Linux instance. Am I reading this right that onboarding your game to Stadia as a developer involved essentially rsyncing data directly to a Linux cloud instance? That's.....

Just the compiled and baked game, not your sources. You still developed on Windows or wherever, the cloud instance was just used for running the game.

Re: CDC File Transfer

#177

Post by loxias »

ww520 wrote: post #167
loxias wrote: post #107

Earlier quoted context omitted.

Hah! I also built a similar storage system, optimized for whole disk images, for work, around 2007!! I used fixed size chunks as well. I called it "data block coalescion", having never heard of anyone else doing so we figured I invented it and we were granted the patent(!). I used it to cram disk images for I think 6 different fresh install configurations onto a single DVD. :D Later on I used it and vmware to build a…

EMC had a disk based deduplication storage at the time. NetAppliance had a competing product. They had patents in the area. I believed that’s in the early 2000’s. One of the household name big techs had an internal product with similar design. ZFS has similar design. Mine was at the block device level. The advantage is you can format it to whatever file system of your choice, with read/write support and deduplication…

> Mine was at the block device level.

Same! :) Originally I wrote it with an interface kinda similar to `tar` -- you add or extract huge blobs to/from what I called a coalesced archive. I could re-image a machine about 8x faster than Norton Ghost.

After $WORK went under, I kept the code and toyed around with it, making it speak NBD so instead of extracting a huge blob from the archive to a destination block device you could also access it directly. I feel like I never Properly solved write support though.

I'm curious, did you think of anything better than refcounting the data blocks and then keeping a list when the count goes to zero, then adding the next unique block to the zero list? That's all I could think of, and it adds at _least_ one additional layer of indirection which I didn't like bc it would have a performance impact.

> EMC had a disk based deduplication storage at the time. NetAppliance had a competing product. They had patents in the area.

I know this _NOW_ but certainly didn't know back then. :) And still doesn't take away the fact that according to the USPTO, "compression via coalescion" is miiiiine. ;-)

Again, I interpret this NOT as evidence of "how clever I am", but as evidence of how silly and broken the patent system is.

Re: CDC File Transfer

#178

Post by ljusten »

greatgib wrote: post #96

I don't understand, you can have rsync use a ssh tunnel directly. Easily. Isn't that enough?

Note that cdc_rsync runs on Windows and syncs to Linux. rsync is a Linux-only tool where you'd have to jump through some hoops to make it work on Windows.

Re: CDC File Transfer

#179

Post by ljusten »

worldsavior wrote: post #157
ljusten wrote: post #153

Earlier quoted context omitted.

That's because most Stadia devs used Windows, but the cloud instances ran on Linux, so devs had to copy their games from Windows to Linux. We're currently adding support for Windows to Windows as well.

But why not make cross platform? If already creating a file transfer program, make it cross platform. What're the complications?

Short answer, we didn't need it. While the code is largely cross-platform, there is some work involved when it gets down to the details.

We are currently working on supporting Windows to Windows. Linux to Linux has lower priority as rsync already provides all functionality, it's just a bit slower on fast connections. On slow connections, rsync and cdc_rsync perform very similarly as the sync speed is dominated by the network.

Post reply on HN