Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bandwidth needs halved by new compression written in Go

arstechnica.com

Post reply on HN
Print view View original

21–30 of 121 posts

Re: Bandwidth needs halved by new compression written in Go

#21

Post by DannyBee »

jgrahamc wrote: post #15
0x0 wrote: post #9

Earlier quoted context omitted.

That sounds interesting, could you elaborate on how it is different from rsync though? "Keep versions synchronized" is a bit vague

The piece in the CloudFlare network and the piece in the customer network are able to keep track of which page versions they each have and so the part in the CloudFlare network sends a request saying "Please do GET /foo and compress it against version X". That means that at request time there's no back-and-forth between the components deciding what compression dictionary to use.

Well, no good binary delta algorithm uses compression dictionaries anyway (since they are binary deltas, not compression algorithms :P), except to compress the newly added strings, which you can't avoid.

Note of course, that relying on the data not being corrupt on the client (which you must if you assume the compression dictionaries are sane) is dangerous. I assume you guys must store some checksum that you compare once to make sure when someone says "i have version 5, delta against this", that they really have a good copy of version 5?

SVN used to what you are suggesting, btw. We only send clients deltas against the versions they already have, and precompute them in some cases :)

Re: Bandwidth needs halved by new compression written in Go

#22

Post by jgrahamc »

DannyBee wrote: post #21
jgrahamc wrote: post #15

Earlier quoted context omitted.

The piece in the CloudFlare network and the piece in the customer network are able to keep track of which page versions they each have and so the part in the CloudFlare network sends a request saying "Please do GET /foo and compress it against version X". That means that at request time there's no back-and-forth between the components deciding what compression dictionary to use.

Well, no good binary delta algorithm uses compression dictionaries anyway (since they are binary deltas, not compression algorithms :P), except to compress the newly added strings, which you can't avoid. Note of course, that relying on the data not being corrupt on the client (which you must if you assume the compression dictionaries are sane) is dangerous. I assume you guys must store some checksum that you compare…

I assume you guys must store some checksum that you compare once to make sure when someone says "i have version 5, delta against this", that they really have a good copy of version 5?

Yes.

Re: Bandwidth needs halved by new compression written in Go

#23

Post by songgao »

corresation wrote: post #6

I was just looking into what SDCH is (an accept-encoding option from Chrome) and it sounds very, very similar: It generates a dictionary and then uses VCDIFF between requests. Is this related somehow?

Railgun is used between CDN and http server, while this one seems to be between browser and http server.

Railgun only requires website to deploy a client, and cooperate with cloudflare. User's client doesn't have to be Chrome or whatever; web server doesn't need to be aware of Railgun. It's transparent to both HTTP clients and HTTP servers. SDCH, however, requires a modification to HTTP/1.1 protocol, which implies changes in both HTTP clients and HTTP servers.

Both are quite promising, though, Railgun is easier to adopt.

Re: Bandwidth needs halved by new compression written in Go

#24

Post by jgrahamc »

rjknight wrote: post #3

The title suggests that there's something unique about Go, either the language or its standard library, that enables bandwidth savings. In fact, Cloudflare have written some software which they claim enables them to reduce their bandwidth, and this software happens to be written in Go. This might be an excellent choice (and I suspect it probably is), but it's not Go per se that is reducing the bandwidth usage.

I agree. The benefit of using Go is that it's fast to write and has good concurrency features. To give you an idea of the size, there are 7,329 lines of Go code in Railgun (including comments) and a 6,602 line test suite.

In the process we've committed various things back to Go itself and at some point I'll write a blog on the whole experience, but one thing that made a big difference was to write a memory recycler so that for commonly created things (in our case []byte buffers) we don't force the garbage collector to keep reaping memory that we then go back and ask for.

The concurrency through communication is trivial to work with once you get the hang of it and being able to write completely sequential code means that it's easy to grok what your own program is doing.

We've hit some deficiencies in the standard library (around HTTP handling) but it's been fairly smooth. And, as the article says, we swapped native crypto for OpenSSL for speed.

The Go tool chain is very nice. Stuff like go fmt, go tool pprof, go vet, go build make working with it smooth.

PS We're hiring.

Re: Bandwidth needs halved by new compression written in Go

#25

Post by vanderZwan »

rjknight wrote: post #3

The title suggests that there's something unique about Go, either the language or its standard library, that enables bandwidth savings. In fact, Cloudflare have written some software which they claim enables them to reduce their bandwidth, and this software happens to be written in Go. This might be an excellent choice (and I suspect it probably is), but it's not Go per se that is reducing the bandwidth usage.

I like Go, but that bothered me too.

From my personal point of view I'm happy they used it though, because it means more people tinkering with and improving the language.

Re: Bandwidth needs halved by new compression written in Go

#27

Post by calinet6 »

dubya wrote: post #8
0x0 wrote: post #4

Earlier quoted context omitted.

It sounds like they reinvented rsync to me?

I think it's more like rsync + git. You have copies of previous versions, and just ask for their hash to figure out which previous version to diff against the current version, then send the diff.

I think that's a good analogy; it's rsync with versioning. Very cool.

Re: Bandwidth needs halved by new compression written in Go

#28

Post by huhtenberg »

DannyBee wrote: post #19
abcd_f wrote: post #12

Earlier quoted context omitted.

> we have more information than rsync does. To what end? Rsync too works off both copies.

rsync is going to perform checksums on blocks to see if the blocks are the same. It transmits these checksums, and where the checksums differ, it deltas the blocks. Note that insertion/deletion in a file can push block boundaries off between two files, causing a problem known as "stream alignment", which can cause your binary delta to be much larger because it doesn't realize the block really shifted 16384 bytes over…

I always thought rsync detected block moves and that's what made it a worthy PhD thesis.

Re: Bandwidth needs halved by new compression written in Go

#29

Post by huhtenberg »

jgrahamc wrote: post #15
0x0 wrote: post #9

Earlier quoted context omitted.

That sounds interesting, could you elaborate on how it is different from rsync though? "Keep versions synchronized" is a bit vague

The piece in the CloudFlare network and the piece in the customer network are able to keep track of which page versions they each have and so the part in the CloudFlare network sends a request saying "Please do GET /foo and compress it against version X". That means that at request time there's no back-and-forth between the components deciding what compression dictionary to use.

For what it's worth, this is fairly standard binary patching approach as used in software updates. I am aware of at least two mainstream titles that do this, and I'd be surprised if Firefox, for example, doesn't push updates this way.

(edit) That's an awesome name by the way. Railgun.

Re: Bandwidth needs halved by new compression written in Go

#30

Post by shanelja »

bitcartel wrote: post #13

The bandwidth reduction is due to use of a binary protocol, not Go. It just so happens the server code is written in Go and C. From the article: “Go is very light,” he said, “and it has fundamental support for concurrent programming. And it’s surprisingly stable for a young language. The experience has been extremely good—there have been no problems with deadlocks or pointer exceptions.” But the code hit a bit of a p…

On another note, it's always nice to see such an influential part of the HN community giving quotes for sites like this - not only does it make me a little proud to be associated with any of you, it makes me more hopeful for the chances of my future that I can call myself one of us.
Post reply on HN