CDC File Transfer • Hacker News Board

Re: CDC File Transfer

#101

Post by zmmmmm » Mon, Jan 09, 2023, 4:22 AM UTC

josephg wrote: post #57

I'm really impressed with the readme file in this repository. Its a master class in effective documentation. Just look at the structure: - 1-2 sentence summary of the project - History (why we built this and didn't just use existing tools) - What we've built, including what it does and how it compares to other tools (rsync in this case). - How to install and use it And the whole thing is full of images and animations…

The documentation in the code itself is pretty great as well:

https://github.com/google/cdc-file-transfer/blob/main/fastcd...

Re: CDC File Transfer

#102

Post by manigandham » Mon, Jan 09, 2023, 4:25 AM UTC

punnerud wrote: post #31

With a small bit of tweaking this could be used for syncing live SQLite databases (probably)

You can use Litestream which is purpose built for streaming SQLite database replicas: https://litestream.io/

Re: CDC File Transfer

#103

Post by manigandham » Mon, Jan 09, 2023, 4:26 AM UTC

raverbashing wrote: post #84

The real question is, why is this not part of the base internet protocols already? Only goes to show how TCP is not as reliable as it should

What does this have to do with TCP being reliable?

Re: CDC File Transfer

#104

Post by TaylorAlexander » Mon, Jan 09, 2023, 4:31 AM UTC

adastra22 wrote: post #46

dleslie wrote: post #45

> However, this was impractical, especially with the shift to working from home during the pandemic with sub-par internet connections. scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression. I didn't appreciate the scope of this problem until a friend of mine visited from the valley. I live in semirural Canada, and…

> I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience I think you took for granted the mere availability of fiber as an option. 30Mbps DSL is the best option I have, other than Starlink. And I live in San Jose!

Yep! I lived all over the Bay Area for the last 20 years and I never had the option for fiber until I moved to Oakland two months ago. It’s only $40 a month for symmetric gigabit fiber. But before I simply had no option for it.

Re: CDC File Transfer

#105

Post by wankle » Mon, Jan 09, 2023, 4:38 AM UTC

If it's for Linux then will we finally have a Google Drive Linux Client?

Re: CDC File Transfer

#106

Post by winrid » Mon, Jan 09, 2023, 5:04 AM UTC

So game devs had to port their games to Linux to run on Stadia servers? That seems like a huge bottleneck...

Re: CDC File Transfer

#107

Post by loxias » Mon, Jan 09, 2023, 5:14 AM UTC

ww520 wrote: post #49

zX41ZdbW wrote: post #13

Content Defined Chunking is one of my favorite algorithms because it has some "magic" similar to HyperLogLogs, Bloom filters, etc... This algorithm is good to explain to people, to get them inspired by computer science. I usually explain the simplest variant with rolling hashes. It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as…

15 years ago when I built a deduplication file storage system, rolling hash was on the table during design but there were some patents on it. Ended up using fixed size chunking which working less well but still gave incredible storage saving.

Hah! I also built a similar storage system, optimized for whole disk images, for work, around 2007!! I used fixed size chunks as well. I called it "data block coalescion", having never heard of anyone else doing so we figured I invented it and we were granted the patent(!). I used it to cram disk images for I think 6 different fresh install configurations onto a single DVD. :D

Later on I used it and vmware to build a proof of concept for a future model of computing where data was never lost. (it would snapshot the vm and add it to the storage system daily or hourly. look ma, infinite undo at a system level!)

The next version of the algorithm was going to use what I now know is essentially rolling hash (I called it "self delimiting data blocks"), but then the company went under.

> but there were some patents on it.

The patent system is quite silly and internally inconsistent. I'm older now, and suspect someone thought of saving disk space through coalescion of identical blocks before I did in 06/07 but not according to the USPTO!

Re: CDC File Transfer

#108

Post by [deleted] » Mon, Jan 09, 2023, 5:17 AM UTC

brendoncarroll wrote: post #3

FastCDC is the same chunking algorithm used in Got. https://github.com/gotvc/got

[deleted]

Re: CDC File Transfer

#109

Post by disqard » Mon, Jan 09, 2023, 5:24 AM UTC

winrid wrote: post #106

So game devs had to port their games to Linux to run on Stadia servers? That seems like a huge bottleneck...

Yes, this was a core architectural requirement of Stadia (running the games on Linux), as opposed to hosting virtualized Windows machines (which would've allowed serving up Windows-based games).

Whether this specific design choice alone killed Stadia, no one can say -- but it was probably one of the most important factors in its demise.

Re: CDC File Transfer

#110

Post by hrrsn » Mon, Jan 09, 2023, 5:33 AM UTC

girvo wrote: post #86

qup wrote: post #75

Earlier quoted context omitted.

10/1 (Mbps) is the "guarantee" where I'm at. It's pretty rural, so I'm happy to have anything. I would pay $200/month extra for fiber.

I pay $130 AUD a month for 500/50 over HFC. Not ideal price wise but it works well enough, but man the NBN could’ve been so much better. I could go to 1000mbps, but then I’m limited to 25mbps upload which is just terrible

I'm starting the process of shifting to Australia (from NZ), and this is going to be the hardest pill to swallow - I'm paying $83 AU for 900/400 currently!