Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC File Transfer

github.com

Post reply on HN
Print view View original

41–50 of 229 posts

Re: CDC File Transfer

#41

Post by klabb3 »

zX41ZdbW wrote: post #13

Content Defined Chunking is one of my favorite algorithms because it has some "magic" similar to HyperLogLogs, Bloom filters, etc... This algorithm is good to explain to people, to get them inspired by computer science. I usually explain the simplest variant with rolling hashes. It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as…

> It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as Amazon S3 or Google Drive (we need metadata storage about chunks, and the chunks can be deduplicated).

Yes this is truly promising but beware of dragons. Under current legal doctrine, blobs need some form of chain of custody. You can’t just deliver chunks to whomever has a hash (unless you’re decentralized, and you can move this problem to your users). Why? Because this is how bittorrent works, and we all know the legal dangers there. Encryption helps against eavesdropping, but not against an adversary who already has the hash and simply wants to prove you are distributing pirated material or even CSAM. You may be able to circumvent this to shift blame back on the user, in some cases. For instance, say you are re-syncing dangerous goods that you initially uploaded over Dropbox, then Dropbox can probably blame you, even though they are technically distributing. But that requires Dropbox to be reasonably confident that “you” (ie the same legal entity) had those chunks in the first place.

Re: CDC File Transfer

#42

Post by bhawks »

eps wrote: post #27
jws wrote: post #6

Earlier quoted context omitted.

To elaborate, rsync chunks in fixed sizes, so inserting or deleting a few bytes makes all different chunks from that point onward. If instead you chunk based off of local content (conceptually like chunking text into sentences at periods, but its a binary thing on has an upper size limit and lower size limit and I couldn't find the algorithm specification) so that after an insertion or deletion in a small number of b…

The gotcha of "inserting or deleting a few bytes" is not in detection, it's in replicating this discovery to the target copy. Say, we have 1GB file and we detected an extra byte at the head of our local copy. Great, what next? We can't replicate this on the receiving end without recopying the file, which is exactly what happens - rsync recreates target file from pieces of its old copy and differences received from th…

> It doesn't work well for archives (tars, zips).

Technically if you update a zip on the remote machine it'll work fine (the data gets appended in an update and the central directory record is always at the end of the zip.

I recall that tar has no end market at all so you can just append a new entry to it as well and when unpacked it'll overwrite the file from earlier in the archive. So they would work fine with rsync unless the tar is also compressed.

The tradeoff between zip and tar.{gz,xz,z} is that zip entries are compressed in the individual file context whereas in a compressed tar the entire archive is compressed in the same context. This may be a slight win for archives with many small files with similar structure.

Re: CDC File Transfer

#43

Post by userbinator »

stefan_ wrote: post #37
5440 wrote: post #20

I'm currently working on a required CDC (Center for Disease Control) reporting function for a COVID test. For a second I thought this article was going to be extremely helpful.

> USB communications device class (or USB CDC class) is a composite Universal Serial Bus device class. I may add this (and it fits somewhat nicely with file transfer..)

That's what my mind also went to first upon seeing the title.

Re: CDC File Transfer

#44

Post by codetrotter »

elromulous wrote: post #7
Waterluvian wrote: post #4

Maybe I’m projecting (probably) but I detect some subtext in the README’s language. Like a “developers who are irate that Stadia was shut down, wanting to free anything they can for the community” type feeling.

Google / abc has a solid track record of open sourcing shutdown projects. See Makani[1] and iirc also Loon. [1] https://news.ycombinator.com/item?id=24456613

Another example is Google Wave, which was also open sourced after it was shut down.

https://en.wikipedia.org/wiki/Google_Wave

https://incubator.apache.org/projects/wave.html

Apache Wave was retired in 2018, but the source remains at least.

Re: CDC File Transfer

#45

Post by dleslie »

> However, this was impractical, especially with the shift to working from home during the pandemic with sub-par internet connections. scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression.

I didn't appreciate the scope of this problem until a friend of mine visited from the valley. I live in semirural Canada, and they were floored by the speed and low latency of the fibre connection I have.

I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience, or at least ask their employer to fund a better connection.

It caused me to reach out, and I found many wfh peers with terrible connections, low powered laptops, and few second monitors. And proper desks? The exception.

My family is mostly in trades, and so spending a little cash to improve my tools felt like common sense. Apparently it isn't.

Re: CDC File Transfer

#46

Post by adastra22 »

dleslie wrote: post #45

> However, this was impractical, especially with the shift to working from home during the pandemic with sub-par internet connections. scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression. I didn't appreciate the scope of this problem until a friend of mine visited from the valley. I live in semirural Canada, and…

> I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience

I think you took for granted the mere availability of fiber as an option.

30Mbps DSL is the best option I have, other than Starlink. And I live in San Jose!

Re: CDC File Transfer

#47

Post by saagarjha »

klabb3 wrote: post #41
zX41ZdbW wrote: post #13

Content Defined Chunking is one of my favorite algorithms because it has some "magic" similar to HyperLogLogs, Bloom filters, etc... This algorithm is good to explain to people, to get them inspired by computer science. I usually explain the simplest variant with rolling hashes. It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as…

> It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as Amazon S3 or Google Drive (we need metadata storage about chunks, and the chunks can be deduplicated). Yes this is truly promising but beware of dragons. Under current legal doctrine, blobs need some form of chain of custody. You can’t just deliver chunks to whomever has a has…

If you’re distributing CSAM on your blob storage, and someone lets you know, you should probably remove it. This is independent of whether you distribute chunks or the whole file.

Re: CDC File Transfer

#48

Post by ocdtrekkie »

codetrotter wrote: post #44
elromulous wrote: post #7

Earlier quoted context omitted.

Google / abc has a solid track record of open sourcing shutdown projects. See Makani[1] and iirc also Loon. [1] https://news.ycombinator.com/item?id=24456613

Another example is Google Wave, which was also open sourced after it was shut down. https://en.wikipedia.org/wiki/Google_Wave https://incubator.apache.org/projects/wave.html Apache Wave was retired in 2018, but the source remains at least.

Apache Wave is still one-click installable on Sandstorm and occasionally hilarious to share instances of with people. Won't do it here though, would kill my poor Intel NUC, Wave is not exactly performance-friendly.

Re: CDC File Transfer

#49

Post by ww520 »

zX41ZdbW wrote: post #13

Content Defined Chunking is one of my favorite algorithms because it has some "magic" similar to HyperLogLogs, Bloom filters, etc... This algorithm is good to explain to people, to get them inspired by computer science. I usually explain the simplest variant with rolling hashes. It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as…

15 years ago when I built a deduplication file storage system, rolling hash was on the table during design but there were some patents on it. Ended up using fixed size chunking which working less well but still gave incredible storage saving.

Re: CDC File Transfer

#50

Post by linsomniac »

punnerud wrote: post #31

With a small bit of tweaking this could be used for syncing live SQLite databases (probably)

Shouldn't require any tweaking, the sliding window should be able to work on sqlite files just as well as the game content files used in this article. If you want something smarter and very much more sqlite optimized, you probably want to look at litefs. https://github.com/superfly/litefs
Post reply on HN