How does cdc_rsync compare to Courgette for sending a 500MB rebuilt ELF binary where .text and .data now have slightly different offsets ?
CDC File Transfer
71–80 of 229 posts
Re: CDC File Transfer
#72Content Defined Chunking is one of my favorite algorithms because it has some "magic" similar to HyperLogLogs, Bloom filters, etc... This algorithm is good to explain to people, to get them inspired by computer science. I usually explain the simplest variant with rolling hashes. It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as…
> It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as Amazon S3 or Google Drive (we need metadata storage about chunks, and the chunks can be deduplicated). Yes this is truly promising but beware of dragons. Under current legal doctrine, blobs need some form of chain of custody. You can’t just deliver chunks to whomever has a has…
That's an interesting extension of the illegal numbers or coloured bits theories, but we don't really see it used that way in practise. When governments or media industry groups crack down on this stuff, they don't go after everybody that ever had those bits in memory. Maybe that's just for practical reasons, but we've never seen every router in between a buyer and seller get confiscated too as they've been somehow tainted. Honestly this doesn't seem like more than a dystopian mental exercise
https://en.wikipedia.org/wiki/Illegal_number https://ansuz.sooke.bc.ca/entry/23 https://shkspr.mobi/blog/2022/11/illegal-hashes/
Re: CDC File Transfer
#73Earlier quoted context omitted.
> It is interesting what the result will be (average saving on deduplication) if it is applied globally to a large-scale blob storage, such as Amazon S3 or Google Drive (we need metadata storage about chunks, and the chunks can be deduplicated). Yes this is truly promising but beware of dragons. Under current legal doctrine, blobs need some form of chain of custody. You can’t just deliver chunks to whomever has a has…
Isn't the main use of bittorrent for ML and research data? Academic torrents is a wonderful resource and what every developer should be using if they need to provide their neural network weights, training data, etc. How is there any legal problem using bittorrent? It's simply much more tailored for this problem than http. It doesn't make any sense to talk about 'Legal problems' for torrent protocols.
What planet have you been living on? Bittorrent is widely used to distribute copyrighted material - movies, TV shows, games, programs, porn... I'd imagine a large majority of bittorrent traffic worldwide is pirated material, with a small portion being datasets as you describe, and other legally-shared data like actual Linux distros, etc.
Re: CDC File Transfer
#74> However, this was impractical, especially with the shift to working from home during the pandemic with sub-par internet connections. scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression. I didn't appreciate the scope of this problem until a friend of mine visited from the valley. I live in semirural Canada, and…
I spend part of my time in a rural area. Only in the last couple years was fiber even available near my house, and then it's a matter of $300/mo on a 3-year contract for 250mbps symmetric, plus the cost of running last-mile from the road to my house.
Thankfully, Starlink came out, and I was able to get like 50/20 for $110/mo on a month-to-month + $500 fixed.
Re: CDC File Transfer
#75> However, this was impractical, especially with the shift to working from home during the pandemic with sub-par internet connections. scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression. I didn't appreciate the scope of this problem until a friend of mine visited from the valley. I live in semirural Canada, and…
> I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience I think you took for granted the mere availability of fiber as an option. 30Mbps DSL is the best option I have, other than Starlink. And I live in San Jose!
10/1 (Mbps) is the "guarantee" where I'm at. It's pretty rural, so I'm happy to have anything.
I would pay $200/month extra for fiber.
Re: CDC File Transfer
#76Earlier quoted context omitted.
> I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience I think you took for granted the mere availability of fiber as an option. 30Mbps DSL is the best option I have, other than Starlink. And I live in San Jose!
10/1 (Mbps) is the "guarantee" where I'm at. It's pretty rural, so I'm happy to have anything. I would pay $200/month extra for fiber.
It still tickles me when people wonder why Starlink is anything someone would want.
The telcos were all given deals with assurances that they would bring rural America to something resembling what their urban counterparts have, but they've remained stagnant for some time now.
Starlink has finally given those people hope of seeing reasonable connectivity options. I would add in that T-Mobile Home Internet, and the likes, are doing so too, but on not as grand of a scale.
Re: CDC File Transfer
#77Maybe I’m projecting (probably) but I detect some subtext in the README’s language. Like a “developers who are irate that Stadia was shut down, wanting to free anything they can for the community” type feeling.
[deleted]
Re: CDC File Transfer
#78> However, this was impractical, especially with the shift to working from home during the pandemic with sub-par internet connections. scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression. I didn't appreciate the scope of this problem until a friend of mine visited from the valley. I live in semirural Canada, and…
> I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience I think you took for granted the mere availability of fiber as an option. 30Mbps DSL is the best option I have, other than Starlink. And I live in San Jose!
Lol, we have 46Mbps DSL over in Half Moon Bay. My friend in Los Altos sent me his 8Gbps symmetric internet speed test result...
Re: CDC File Transfer
#79sounds like cdc_rsync should be a good replacement for rsync in a generic backup
Imagine an rsync that was A LOT smarter for backup use cases. Detecting files that have been renamed, files that have been compressed, similar files, files that have only been appended to, files that are the same across multiple systems (Oh, yeah, I probably already have a copy of this kernel file from this other host). Basically, deduplication as part of rsync.
BorgBackup has most of what you're looking for, though it doesn't implement CDC and doesn't replicate the files as-is in the backup location (instead using a compressed/deduped/chunked storage format)
Re: CDC File Transfer
#80> However, this was impractical, especially with the shift to working from home during the pandemic with sub-par internet connections. scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression. I didn't appreciate the scope of this problem until a friend of mine visited from the valley. I live in semirural Canada, and…
> I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience I think you took for granted the mere availability of fiber as an option. 30Mbps DSL is the best option I have, other than Starlink. And I live in San Jose!
that's wild, my parents in a pop.~ 1000 rural town in france get 2GB fiber; the next "average" town (60k inhabitants) is 30km away, and the next big town (Toulouse, pop. 460k) is 120km away