Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC File Transfer

github.com

Post reply on HN
Print view View original

161–170 of 229 posts

Re: CDC File Transfer

#161

Post by aembleton »

worldsavior wrote: post #157
ljusten wrote: post #153

Earlier quoted context omitted.

That's because most Stadia devs used Windows, but the cloud instances ran on Linux, so devs had to copy their games from Windows to Linux. We're currently adding support for Windows to Windows as well.

But why not make cross platform? If already creating a file transfer program, make it cross platform. What're the complications?

It does do Linux to Linux as that's covered by the Readme. For anything else, I guess it didn't fill the scope of what was needed for Stadia

Re: CDC File Transfer

#162

Post by flipbrad »

klabb3 wrote: post #91
ketralnis wrote: post #72

Earlier quoted context omitted.

That's an interesting extension of the illegal numbers or coloured bits theories, but we don't really see it used that way in practise. When governments or media industry groups crack down on this stuff, they don't go after everybody that ever had those bits in memory. Maybe that's just for practical reasons, but we've never seen every router in between a buyer and seller get confiscated too as they've been somehow t…

I’m not suggesting the hashes themselves are illegal to possess, but that transferring the bytes corresponding to those hashes is problematic: if both sides are lowly trusted, that puts you at risk as a hoster of that content. This is indeed an issue with IPFS, for instance, where I believe the solutions are “pinning” content that is already vetted by another party, or denylists of “bad bits”. I assume it’s similar t…

Re last para: probably because it's full of very certain, but also quite certainly wrong, statements along the lines of "Under current legal doctrine, blobs need some form of chain of custody." Citation needed.

Re: CDC File Transfer

#163

Post by aembleton »

jamespo wrote: post #156
ljusten wrote: post #153

Earlier quoted context omitted.

That's because most Stadia devs used Windows, but the cloud instances ran on Linux, so devs had to copy their games from Windows to Linux. We're currently adding support for Windows to Windows as well.

And linux to linux?

We also ran the experiment with the native Linux rsync, i.e syncing Linux to Linux, to rule out issues with Cygwin. Linux rsync performed on average 35% worse than Cygwin rsync, which can be attributed to CPU differences.

Source: https://github.com/google/cdc-file-transfer

Re: CDC File Transfer

#164

Post by capableweb »

_ea1k wrote: post #82

Looks interesting, but when did we start saying "synching" instead of "syncing"? I mean, its literally based on the idea behind rsync. Why the sudden need to add an "h"?

If by "we" you mean the internet, it seems most people still use "syncing" rather than "synching": https://trends.google.com/trends/explore?date=all&q=syncing,...

Still, "syncing" is supposed to be shorthand for "synchronizing", which has the "h" you feel comes out of nowhere. So I guess both makes sense, but I don't use that form myself nor have seen anyone else use it in the wild.

Re: CDC File Transfer

#165

Post by dan-robertson »

Topgamer7 wrote: post #134
dleslie wrote: post #60

Earlier quoted context omitted.

You're not wrong. It's amazing how poor access can be. I live in a tiny town on Vancouver Island and I have gigabit symmetric, for a very reasonable price. When I lived in Vancouver that simply wasn't an option.

The Canadian government gives grants to telecoms to install fibre. And I think there is some relation to signing up end users too but I can't recall.

I thought the USG did a similar thing but maybe didn’t require the installed fibre actually be used /connected to the cabinets?

Re: CDC File Transfer

#166

Post by ljusten »

eps wrote: post #27
jws wrote: post #6

Earlier quoted context omitted.

To elaborate, rsync chunks in fixed sizes, so inserting or deleting a few bytes makes all different chunks from that point onward. If instead you chunk based off of local content (conceptually like chunking text into sentences at periods, but its a binary thing on has an upper size limit and lower size limit and I couldn't find the algorithm specification) so that after an insertion or deletion in a small number of b…

The gotcha of "inserting or deleting a few bytes" is not in detection, it's in replicating this discovery to the target copy. Say, we have 1GB file and we detected an extra byte at the head of our local copy. Great, what next? We can't replicate this on the receiving end without recopying the file, which is exactly what happens - rsync recreates target file from pieces of its old copy and differences received from th…

For reference, here's the paper that describes the in-place update algo in rsync: https://www.usenix.org/legacy/events/usenix03/tech/freenix03.... I haven't looked into it more deeply, but I think it's possible to apply the same idea to variable sized chunks.

Also, most modern compression tools have an "rsyncable" option that makes the archives play more nicely with rsync.

Re: CDC File Transfer

#167

Post by ww520 »

loxias wrote: post #107
ww520 wrote: post #49

Earlier quoted context omitted.

15 years ago when I built a deduplication file storage system, rolling hash was on the table during design but there were some patents on it. Ended up using fixed size chunking which working less well but still gave incredible storage saving.

Hah! I also built a similar storage system, optimized for whole disk images, for work, around 2007!! I used fixed size chunks as well. I called it "data block coalescion", having never heard of anyone else doing so we figured I invented it and we were granted the patent(!). I used it to cram disk images for I think 6 different fresh install configurations onto a single DVD. :D Later on I used it and vmware to build a…

EMC had a disk based deduplication storage at the time. NetAppliance had a competing product. They had patents in the area. I believed that’s in the early 2000’s. One of the household name big techs had an internal product with similar design. ZFS has similar design.

Mine was at the block device level. The advantage is you can format it to whatever file system of your choice, with read/write support and deduplication just works.

Re: CDC File Transfer

#168

Post by ClassyJacket »

dleslie wrote: post #45

> However, this was impractical, especially with the shift to working from home during the pandemic with sub-par internet connections. scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression. I didn't appreciate the scope of this problem until a friend of mine visited from the valley. I live in semirural Canada, and…

Nobody is avoiding fibre because it costs extra, we don't have fibre because it isn't offered. We're not as stupid as you're accusing us of being.

Re: CDC File Transfer

#169

Post by c7b »

skies457 wrote: post #158
c7b wrote: post #130

Slightly OT, but I like the schematic gifs used in the Readme.md (pretty amazing doc overall!) like this one [0]. Does anyone have suggestions what tools they might have used (or might be used in general) to create those? [0] https://github.com/google/cdc-file-transfer/blob/main/docs/l...

D2 ( https://d2lang.com/tour/intro/ ) might be the tool you are looking for, but it doesn't handle animation as far as I know

That looks useful, thanks!

Re: CDC File Transfer

#170

Post by klabb3 »

flipbrad wrote: post #162
klabb3 wrote: post #91

Earlier quoted context omitted.

I’m not suggesting the hashes themselves are illegal to possess, but that transferring the bytes corresponding to those hashes is problematic: if both sides are lowly trusted, that puts you at risk as a hoster of that content. This is indeed an issue with IPFS, for instance, where I believe the solutions are “pinning” content that is already vetted by another party, or denylists of “bad bits”. I assume it’s similar t…

Re last para: probably because it's full of very certain, but also quite certainly wrong, statements along the lines of "Under current legal doctrine, blobs need some form of chain of custody." Citation needed.

I can see how that’s overly assuming. Thanks for being candid.
Post reply on HN