Bandwidth needs halved by new compression written in Go
arstechnica.com
Bandwidth needs halved by new compression written in Go
1–10 of 121 posts
Re: Bandwidth needs halved by new compression written in Go
#2Go is used, sure, but the cool part about this is the binary Railgun protocol. Really smart. Send only file hashes and binary diffs back and forth, do a little extra computation to figure out the changes, but only send the absolute minimum data you need to the CDN. That's just smart, and frankly, I hope other CDNs have been doing this already, because at any high volume it seems to be an obvious solution.
So that brings up the question—is this just something CloudFlare is announcing for the PR, or is it actually innovative?
Re: Bandwidth needs halved by new compression written in Go
#3The title suggests that there's something unique about Go, either the language or its standard library, that enables bandwidth savings. In fact, Cloudflare have written some software which they claim enables them to reduce their bandwidth, and this software happens to be written in Go. This might be an excellent choice (and I suspect it probably is), but it's not Go per se that is reducing the bandwidth usage.
Re: Bandwidth needs halved by new compression written in Go
#4Go is used, sure, but the cool part about this is the binary Railgun protocol. Really smart. Send only file hashes and binary diffs back and forth, do a little extra computation to figure out the changes, but only send the absolute minimum data you need to the CDN. That's just smart, and frankly, I hope other CDNs have been doing this already, because at any high volume it seems to be an obvious solution. So that bri…
It sounds like they reinvented rsync to me?
Re: Bandwidth needs halved by new compression written in Go
#5Go is used, sure, but the cool part about this is the binary Railgun protocol. Really smart. Send only file hashes and binary diffs back and forth, do a little extra computation to figure out the changes, but only send the absolute minimum data you need to the CDN. That's just smart, and frankly, I hope other CDNs have been doing this already, because at any high volume it seems to be an obvious solution. So that bri…
It sounds like they reinvented rsync to me?
No, because we have more information than rsync does. We own both ends of the connection and can keep versions synchronized.
Re: Bandwidth needs halved by new compression written in Go
#6I was just looking into what SDCH is (an accept-encoding option from Chrome) and it sounds very, very similar: It generates a dictionary and then uses VCDIFF between requests. Is this related somehow?
Re: Bandwidth needs halved by new compression written in Go
#7Could be very cool. I couldn't get through the article because it read like a press release. Maybe if someone who hasn't been spoon-fed the story reports on it, I'll take notice.
Re: Bandwidth needs halved by new compression written in Go
#8Go is used, sure, but the cool part about this is the binary Railgun protocol. Really smart. Send only file hashes and binary diffs back and forth, do a little extra computation to figure out the changes, but only send the absolute minimum data you need to the CDN. That's just smart, and frankly, I hope other CDNs have been doing this already, because at any high volume it seems to be an obvious solution. So that bri…
It sounds like they reinvented rsync to me?
I think it's more like rsync + git. You have copies of previous versions, and just ask for their hash to figure out which previous version to diff against the current version, then send the diff.
Re: Bandwidth needs halved by new compression written in Go
#9Earlier quoted context omitted.
It sounds like they reinvented rsync to me?
No, because we have more information than rsync does. We own both ends of the connection and can keep versions synchronized.
That sounds interesting, could you elaborate on how it is different from rsync though? "Keep versions synchronized" is a bit vague
Re: Bandwidth needs halved by new compression written in Go
#10Go is used, sure, but the cool part about this is the binary Railgun protocol. Really smart. Send only file hashes and binary diffs back and forth, do a little extra computation to figure out the changes, but only send the absolute minimum data you need to the CDN. That's just smart, and frankly, I hope other CDNs have been doing this already, because at any high volume it seems to be an obvious solution. So that bri…
It sounds like they reinvented rsync to me?
Not sure why you're downvoted. This is basically what rsync is: a smart algorithm for doing rolling checksums and only sending diffs.