Live data from Hacker News

CDC File Transfer

github.com

121–130 of 229 posts

Re: CDC File Transfer

#121
post #94
post #73

Earlier quoted context omitted.

What planet have you been living on? Bittorrent is widely used to distribute copyrighted material - movies, TV shows, games, programs, porn... I'd imagine a large majority of bittorrent traffic worldwide is pirated material, with a small portion being datasets as you describe, and other legally-shared data like actual Linux distros, etc.

Torrents are no longer main source of copyrighted materials, at least for shows and movies. There is a bunch of illegal services that provide Netflix like experience against pirated content.

Don't these services usually use torrents under the hood though? Thinking about stremio and popcorntime.

Re: CDC File Transfer

#122
post #112

This might be a dumb question.. but when I read this "scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression." my thinking was: If you want to send diffs.. why not just use git? It does compression and chunks and all that. Maybe the defaults perform poorly for binary files? But couldn't you fix that with a custom d…

Even if one tries to repurpose git for it, git doesn't work well with diffs in binaries

Re: CDC File Transfer

#123

Always interesting to see the sort of tangential projects that get rolled into the budget of larger projects at companies like Google. Funding the development of yet another file transfer utility, interesting as this one is, definitely wasn't necessary.

> Funding the development of yet another file transfer utility, interesting as this one is, definitely wasn't necessary.

Interesting you say that as I thought they explained their motivation pretty clearly. Is there a tool they missed that already does the job?

Re: CDC File Transfer

#125
> At Stadia, game developers had access to Linux cloud instances to run games. Most developers wrote their games on Windows, though. Therefore, they needed a way to make them available on the remote Linux instance.

Am I reading this right that onboarding your game to Stadia as a developer involved essentially rsyncing data directly to a Linux cloud instance?

That's.....

Re: CDC File Transfer

#126
post #123

Always interesting to see the sort of tangential projects that get rolled into the budget of larger projects at companies like Google. Funding the development of yet another file transfer utility, interesting as this one is, definitely wasn't necessary.

> Funding the development of yet another file transfer utility, interesting as this one is, definitely wasn't necessary. Interesting you say that as I thought they explained their motivation pretty clearly. Is there a tool they missed that already does the job?

I think most companies would have used an off the shelf solution and expected devs to simply deal with the inefficiency.

Re: CDC File Transfer

#127
post #95

Earlier quoted context omitted.

> I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience I think you took for granted the mere availability of fiber as an option. 30Mbps DSL is the best option I have, other than Starlink. And I live in San Jose!

Bay Area broadband is surprisingly bad, and doesn’t fit the outside assumptions of the “Silicon Valley”.

It’s because of Silicon Valley I think. We all got broadband with the first generation of technology in the 90’s. Since then, there has been so much government subsidies for rolling out internet infrastructure that no telco will invest in upgrades unless there is government money to pay for it. But most of those subsidies only apply to new deployments, not upgrades. We don’t qualify because we already have “broadband” (which here just means faster than dialup). So nobody is willing to pony up the cash for fiber-to-the-home deployments.

Re: CDC File Transfer

#128
post #112

This might be a dumb question.. but when I read this "scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression." my thinking was: If you want to send diffs.. why not just use git? It does compression and chunks and all that. Maybe the defaults perform poorly for binary files? But couldn't you fix that with a custom d…

They are separate problems and not really comparable, git is a higher level tool. The diffing in git is for humans to compare code, and as such it is actually computed on the fly and not stored anywhere (unless you use git patch). Similarly, git does not provide any mechanism for sending/receiving files, it re-uses existing solutions like HTTP and SSH. This tool exists to send/receive files, and the diffing is an imp…

Oh right, git doesn't store diffs. So each time you push/pull, it fetches/uploads the whole file.. for every commit?

Damn..

Then I guess even using this under the hood would be essentially a rewrite or how git fundamentally works internally

Re: CDC File Transfer

#129
post #106

So game devs had to port their games to Linux to run on Stadia servers? That seems like a huge bottleneck...

Yes, this was a core architectural requirement of Stadia (running the games on Linux), as opposed to hosting virtualized Windows machines (which would've allowed serving up Windows-based games). Whether this specific design choice alone killed Stadia, no one can say -- but it was probably one of the most important factors in its demise.

For me, the two most important aspects were: 1) it's made by Google so it will probably be shut down soon, and 2) if you already have any hardware which can play games, Stadia is strictly more expensive than buying and playing games locally. That's before I even got to the point of considering the size of its game library or the technical qualities of the service.

I play all my games on Linux, and in general, there are extremely few games I haven't gotten to run. Many just work out of the box through Proton, games not on Steam usually work out of the box with Lutris. This suggests to me that the Linux thing wasn't necessarily a big issue, if Google was willing to put in some work to get existing Windows games to work. AFAIK, they didn't do this and instead required games to run natively on Linux, which would significantly reduce the size of their games library (after all, why would a company invest time into porting their games to a Google product that's going to be shut down?).

Whether the US is even a market that's ready for game streaming is another question. Are there enough people with a high quality, low latency Internet connection that's close enough to a Stadia data center, who are interested in playing demanding games, but don't have the money for hardware yet can afford a monthly subscription on top of a high per-game price? Maybe it would be interesting if it worked like other subscription services and the subscription itself gave you access to games. I think Spotify would have flopped if you had to buy albums at their retail price in addition to the monthly subscription.

Post reply on HN