Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC File Transfer

github.com

Post reply on HN
Print view View original

141–150 of 229 posts

Re: CDC File Transfer

#141

Post by Tor3 »

adastra22 wrote: post #127
BooneJS wrote: post #95

Earlier quoted context omitted.

Bay Area broadband is surprisingly bad, and doesn’t fit the outside assumptions of the “Silicon Valley”.

It’s because of Silicon Valley I think. We all got broadband with the first generation of technology in the 90’s. Since then, there has been so much government subsidies for rolling out internet infrastructure that no telco will invest in upgrades unless there is government money to pay for it. But most of those subsidies only apply to new deployments, not upgrades. We don’t qualify because we already have “broadband…

It's surprising that it would still need government subsidies to start with. I'm not really into the details of such thing that much, but around here (northern Europe) we're basically getting fiber everywhere, first in cities but later also in very rural and (by European standards) low-population areas, and there are no subsidies involved. Private companies compete and goes to communities (in rural areas) or neighborhoods (in city- or city-like areas) and try to drum up interest, and if there's enough interest they start digging. And in densely populated areas it's a no-brainer, they just do it. Due to competition (just more than one supplier and you have the competition) it's kind of "got to start this area before the competitor does.."

The fiber is also used for TV etc., with several providers on the same fiber, so I imagine that the fiber provider gets income from the competing TV providers as well. Could be part of the why. As for myself, I only need and only pay for actual internet.

I've had 1Gb/1Gb fiber for many years now, and lately fiber arrives in the most unexpected areas (long distances, few residents). It (the deployment, not the monthly) used to be more costly, but it's not anymore. And no public money involved. I know that it used to be, as an experiment, a couple of decades or more ago, but only in certain areas.

Re: CDC File Transfer

#142

Post by ljusten »

deathanatos wrote: post #116
geokon wrote: post #112

This might be a dumb question.. but when I read this "scp always copies full files, there is no "delta mode" to copy only the things that changed, it is slow for many small files, and there is no fast compression." my thinking was: If you want to send diffs.. why not just use git? It does compression and chunks and all that. Maybe the defaults perform poorly for binary files? But couldn't you fix that with a custom d…

I'm sort of in the same boat, but with the sentence, > To help this situation, we developed two tools, cdc_rsync Why not use rsync? (It does seem they ended up faster than rsync, so perhaps that's "why", but that seems more like a post-hoc justification.) The "we use variable chunk windows" bit is intriguing, but the example GIF sort of just pre-supposes that the local & remote chunks match up. That could have happen…

>> Or, how is it that the local manages to be clairvoyant enough to choose the same windows?

Yes! In content defined chunking, the chunk boundaries depend on the content, in our case a 64 byte window. If the local and the remote files have the same 64 byte sequence anywhere, and that 64 byte sequence has some magic pattern of 0s and 1s, they will both have chunk boundaries there. A chunk is the range of data between two chunk boundaries, so if N consecutive chunk boundaries match, then N-1 consecutive chunks match.

Re: CDC File Transfer

#143

Post by raverbashing »

kccqzy wrote: post #89
raverbashing wrote: post #84

The real question is, why is this not part of the base internet protocols already? Only goes to show how TCP is not as reliable as it should

There are many older and more venerable file transfer protocols, like FTP or TFTP. Add encryption and then you have SCP and SFTP. SFTP is the default protocol used by the scp command, and TFTP is still often used to communicate with routers and access points and similar equipment. And I'm not sure what that has anything to do with TCP not as reliable. TCP has real problems on the modern Internet but the first part do…

And none of those protocols keeps the connection alive if you unplug your network cable and plug it back on really quickly

Might I say, even if you get the same IP back from the router

Re: CDC File Transfer

#145

Post by int_19h »

mindwok wrote: post #117
deathanatos wrote: post #116

Earlier quoted context omitted.

I'm sort of in the same boat, but with the sentence, > To help this situation, we developed two tools, cdc_rsync Why not use rsync? (It does seem they ended up faster than rsync, so perhaps that's "why", but that seems more like a post-hoc justification.) The "we use variable chunk windows" bit is intriguing, but the example GIF sort of just pre-supposes that the local & remote chunks match up. That could have happen…

I was wondering the same thing. It isn't explicit in the write-up but it's because rsync does not have a native Windows implementation, only rsync under Cygwin, so they developed this to achieve the same thing except it turned out faster than rsync.

MSYS2 has rsync. Unlike Cygwin, I would consider that native.

Re: CDC File Transfer

#146

Post by tmchow »

1. I thought this was going to be something about the Center for Disease Control

2. This is a really neat project, and super expensive ashes to rise from Stadia.

3. This is targeted at Windows to Linux, but given the speed advantages that it has over rsync couldn’t this be Linux to Linux? That said, I’d be afraid to use this over rsync just from the body of experience and knowledge that exists about rsync.

Re: CDC File Transfer

#147

Post by ljusten »

mjevans wrote: post #21
sigmonsays wrote: post #2

can we gut rsync and implement this? rsync doesn't even have a great protocol specification... maybe it's a new tool...

I'd like additional info about the Content Defined Chunking part of the specification. How is the content defined? Where I'd try to begin is with a single pass that looks for runs of 'null' ('\0') bytes, even one long, as potential boundary ends. During that pass also look for 'magic signatures' for known stream types like already compressed content streams (all the more so to just not try compressing anyway). The CD…

In a nutshell, the algorithm computes

  uint64_t hash = 0;
  uint64_t magic_pattern = 0b001000010000100001000...;
  for (size_t n = 0; n
In practice, there's more bells and whistles, but that's the gist of it. By tweaking the numbers of 1's in magic_pattern you can influence the average chunk size (distance between two boundaries). With every additional 1, your chunk size halves. There's no special handling of compressed file types. You'd probably want to do that at a much higher level, e.g. just check for extensions.

Re: CDC File Transfer

#148

Post by haarts »

hot_gril wrote: post #70

> Born from the ashes of Stadia Love it. Those were some very expensive ashes. I hope more comes from them too.

I take heart when reading about recovered parts of expensive ashes.

But I firmly believe that no project is wasted especially one that pushed the boundaries of what is commonly done. All the people working there learnt something. They might apply their new found knowledge to other fields or future carriers. The money invested is not lost but converted into bigger brains.

I look at VC money in much the same way. It's not great when a startup fails. But a lot is learnt.

Re: CDC File Transfer

#149

Post by r4kbY44D »

c7b wrote: post #130

Slightly OT, but I like the schematic gifs used in the Readme.md (pretty amazing doc overall!) like this one [0]. Does anyone have suggestions what tools they might have used (or might be used in general) to create those? [0] https://github.com/google/cdc-file-transfer/blob/main/docs/l...

ImageMagick does the trick, for example `convert -delay 20 -loop 0 *.png out.gif`

Re: CDC File Transfer

#150

Post by c7b »

ljusten wrote: post #138
c7b wrote: post #130

Slightly OT, but I like the schematic gifs used in the Readme.md (pretty amazing doc overall!) like this one [0]. Does anyone have suggestions what tools they might have used (or might be used in general) to create those? [0] https://github.com/google/cdc-file-transfer/blob/main/docs/l...

We used an internal Chrome extension to capture the gifs. There also seem to be some externally available GIF capturing tools, like https://chrome.google.com/webstore/detail/chrome-capture-scr... (disclaimer: I haven't tried those). We then used https://ezgif.com/optimize to remove dupe/unwanted frames, tweak timings and compress the gif. The actual content was done in Google Slides.

Awesome, thanks for replying here! If I may be greedy: I suppose you used something else for capturing the command prompt gif?
Post reply on HN