Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC File Transfer

github.com

Post reply on HN
Print view View original

91–100 of 229 posts

Re: CDC File Transfer

#91

Post by klabb3 »

ketralnis wrote: post #72
klabb3 wrote: post #41

Earlier quoted context omitted.

> It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as Amazon S3 or Google Drive (we need metadata storage about chunks, and the chunks can be deduplicated). Yes this is truly promising but beware of dragons. Under current legal doctrine, blobs need some form of chain of custody. You can’t just deliver chunks to whomever has a has…

That's an interesting extension of the illegal numbers or coloured bits theories, but we don't really see it used that way in practise. When governments or media industry groups crack down on this stuff, they don't go after everybody that ever had those bits in memory. Maybe that's just for practical reasons, but we've never seen every router in between a buyer and seller get confiscated too as they've been somehow t…

I’m not suggesting the hashes themselves are illegal to possess, but that transferring the bytes corresponding to those hashes is problematic: if both sides are lowly trusted, that puts you at risk as a hoster of that content. This is indeed an issue with IPFS, for instance, where I believe the solutions are “pinning” content that is already vetted by another party, or denylists of “bad bits”. I assume it’s similar to any other clearnet hosting. Btw, I make zero value judgments about all of that.

Off topic: I see downvotes on my parent comment, please let me know if I said something bad to help me improve.

Re: CDC File Transfer

#92

Post by klabb3 »

saagarjha wrote: post #47
klabb3 wrote: post #41

Earlier quoted context omitted.

> It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as Amazon S3 or Google Drive (we need metadata storage about chunks, and the chunks can be deduplicated). Yes this is truly promising but beware of dragons. Under current legal doctrine, blobs need some form of chain of custody. You can’t just deliver chunks to whomever has a has…

If you’re distributing CSAM on your blob storage, and someone lets you know, you should probably remove it. This is independent of whether you distribute chunks or the whole file.

I think for piracy/DMCA it’s enough to simply remove it. As for CSAM or more serious stuff, I don’t know if that’s enough? Does section 230 cover that? Is there a difference between being a company and an individual?

Re: CDC File Transfer

#93

Post by fnordpiglet »

consumer451 wrote: post #83
adastra22 wrote: post #46

Earlier quoted context omitted.

> I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience I think you took for granted the mere availability of fiber as an option. 30Mbps DSL is the best option I have, other than Starlink. And I live in San Jose!

I am sitting in a farming village in central EU and I have 900mbps fiber, as of last year. My family lives in old South Seattle neighborhoods and has excellent fiber. I had something like 100mpbs co-axial 21 years ago in West Seattle. What is the problem with San Jose?

I think people have been asking that question for generations.

Re: CDC File Transfer

#94

Post by AlfeG »

vonseel wrote: post #73
chaxor wrote: post #58

Earlier quoted context omitted.

Isn't the main use of bittorrent for ML and research data? Academic torrents is a wonderful resource and what every developer should be using if they need to provide their neural network weights, training data, etc. How is there any legal problem using bittorrent? It's simply much more tailored for this problem than http. It doesn't make any sense to talk about 'Legal problems' for torrent protocols.

What planet have you been living on? Bittorrent is widely used to distribute copyrighted material - movies, TV shows, games, programs, porn... I'd imagine a large majority of bittorrent traffic worldwide is pirated material, with a small portion being datasets as you describe, and other legally-shared data like actual Linux distros, etc.

Torrents are no longer main source of copyrighted materials, at least for shows and movies. There is a bunch of illegal services that provide Netflix like experience against pirated content.

Re: CDC File Transfer

#95

Post by BooneJS »

adastra22 wrote: post #46
dleslie wrote: post #45

> However, this was impractical, especially with the shift to working from home during the pandemic with sub-par internet connections. scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression. I didn't appreciate the scope of this problem until a friend of mine visited from the valley. I live in semirural Canada, and…

> I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience I think you took for granted the mere availability of fiber as an option. 30Mbps DSL is the best option I have, other than Starlink. And I live in San Jose!

Bay Area broadband is surprisingly bad, and doesn’t fit the outside assumptions of the “Silicon Valley”.

Re: CDC File Transfer

#97

Post by Thorrez »

Mortiffer wrote: post #39

Lame move google, CDC = Change Data Capture .

The acronym CDC meaning Content-Defined Chunking wasn't invented by Google I don't think. Here's the 2016 paper introducing the FastCDC algorithm[1]. Disclosure: I work at Google, but not on anything related to this.

[1] https://www.usenix.org/system/files/conference/atc16/atc16-p...

Re: CDC File Transfer

#100

Post by [deleted] »

_ea1k wrote: post #82

Looks interesting, but when did we start saying "synching" instead of "syncing"? I mean, its literally based on the idea behind rsync. Why the sudden need to add an "h"?

[deleted]
Post reply on HN