Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC File Transfer

github.com

Post reply on HN
Print view View original

111–120 of 229 posts

Re: CDC File Transfer

#111

Post by linsomniac »

RandomBK wrote: post #79
linsomniac wrote: post #53

Earlier quoted context omitted.

Imagine an rsync that was A LOT smarter for backup use cases. Detecting files that have been renamed, files that have been compressed, similar files, files that have only been appended to, files that are the same across multiple systems (Oh, yeah, I probably already have a copy of this kernel file from this other host). Basically, deduplication as part of rsync.

BorgBackup has most of what you're looking for, though it doesn't implement CDC and doesn't replicate the files as-is in the backup location (instead using a compressed/deduped/chunked storage format)

For single system backups I've switched to using restic and it's been pretty great. I don't trust Borg, a couple years ago I tried doing a recovery using it and ran into some unicode issue in, I believe, a filename, and I couldn't track down exactly what file it was or get any files backed up after that file in the archive. I ended up using another backup I had.

For my multiple backups to a backup host where I'm using rsync, restic really doesn't work (having 100+ systems backed up to the same destination).

Re: CDC File Transfer

#112

Post by geokon »

This might be a dumb question.. but when I read this

"scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression."

my thinking was: If you want to send diffs.. why not just use git?

It does compression and chunks and all that. Maybe the defaults perform poorly for binary files? But couldn't you fix that with a custom diff algo? (if it's somehow not appropriate, it'd still be nice to port whatever secret sauce they use here to git..)

Re: CDC File Transfer

#113

Post by petesergeant »

zX41ZdbW wrote: post #13

Content Defined Chunking is one of my favorite algorithms because it has some "magic" similar to HyperLogLogs, Bloom filters, etc... This algorithm is good to explain to people, to get them inspired by computer science. I usually explain the simplest variant with rolling hashes. It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as…

https://restic.net/blog/2015-09-12/restic-foundation1-cdc/

Re: CDC File Transfer

#114

Post by verdverm »

disqard wrote: post #109
winrid wrote: post #106

So game devs had to port their games to Linux to run on Stadia servers? That seems like a huge bottleneck...

Yes, this was a core architectural requirement of Stadia (running the games on Linux), as opposed to hosting virtualized Windows machines (which would've allowed serving up Windows-based games). Whether this specific design choice alone killed Stadia, no one can say -- but it was probably one of the most important factors in its demise.

Input lag and pixelated graphics is why I quit, I had gigabit fiber at the time

Re: CDC File Transfer

#116

Post by deathanatos »

geokon wrote: post #112

This might be a dumb question.. but when I read this "scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression." my thinking was: If you want to send diffs.. why not just use git? It does compression and chunks and all that. Maybe the defaults perform poorly for binary files? But couldn't you fix that with a custom d…

I'm sort of in the same boat, but with the sentence,

> To help this situation, we developed two tools, cdc_rsync

Why not use rsync?

(It does seem they ended up faster than rsync, so perhaps that's "why", but that seems more like a post-hoc justification.)

The "we use variable chunk windows" bit is intriguing, but the example GIF sort of just pre-supposes that the local & remote chunks match up. That could have happened in the rsync case/GIF, but that wasn't the case considered, so it's an oranges/apples comparison. (Or, how is it that the local manages to be clairvoyant enough to choose the same windows?)

Re: CDC File Transfer

#117

Post by mindwok »

deathanatos wrote: post #116
geokon wrote: post #112

This might be a dumb question.. but when I read this "scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression." my thinking was: If you want to send diffs.. why not just use git? It does compression and chunks and all that. Maybe the defaults perform poorly for binary files? But couldn't you fix that with a custom d…

I'm sort of in the same boat, but with the sentence, > To help this situation, we developed two tools, cdc_rsync Why not use rsync? (It does seem they ended up faster than rsync, so perhaps that's "why", but that seems more like a post-hoc justification.) The "we use variable chunk windows" bit is intriguing, but the example GIF sort of just pre-supposes that the local & remote chunks match up. That could have happen…

I was wondering the same thing. It isn't explicit in the write-up but it's because rsync does not have a native Windows implementation, only rsync under Cygwin, so they developed this to achieve the same thing except it turned out faster than rsync.

Re: CDC File Transfer

#118

Post by LarryMullins »

Always interesting to see the sort of tangential projects that get rolled into the budget of larger projects at companies like Google. Funding the development of yet another file transfer utility, interesting as this one is, definitely wasn't necessary.

Re: CDC File Transfer

#119

Post by mindwok »

geokon wrote: post #112

This might be a dumb question.. but when I read this "scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression." my thinking was: If you want to send diffs.. why not just use git? It does compression and chunks and all that. Maybe the defaults perform poorly for binary files? But couldn't you fix that with a custom d…

They are separate problems and not really comparable, git is a higher level tool. The diffing in git is for humans to compare code, and as such it is actually computed on the fly and not stored anywhere (unless you use git patch). Similarly, git does not provide any mechanism for sending/receiving files, it re-uses existing solutions like HTTP and SSH.

This tool exists to send/receive files, and the diffing is an implementation detail used to achieve a high level of performance. It would make more sense for git to use this library under the hood as you mentioned.

Re: CDC File Transfer

#120

Post by loxias »

AlfeG wrote: post #94
vonseel wrote: post #73

Earlier quoted context omitted.

What planet have you been living on? Bittorrent is widely used to distribute copyrighted material - movies, TV shows, games, programs, porn... I'd imagine a large majority of bittorrent traffic worldwide is pirated material, with a small portion being datasets as you describe, and other legally-shared data like actual Linux distros, etc.

Torrents are no longer main source of copyrighted materials, at least for shows and movies. There is a bunch of illegal services that provide Netflix like experience against pirated content.

Now I feel old, using bittorrent and soulseek.
Post reply on HN