Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bandwidth needs halved by new compression written in Go

arstechnica.com

Post reply on HN
Print view View original

11–20 of 121 posts

Re: Bandwidth needs halved by new compression written in Go

#11

Post by jgrahamc »

corresation wrote: post #6

I was just looking into what SDCH is (an accept-encoding option from Chrome) and it sounds very, very similar: It generates a dictionary and then uses VCDIFF between requests. Is this related somehow?

Vaguely. Both Railgun and SDHC work by compressing web pages against an external dictionary. In SDHC the dictionary must be generated (somehow), and it is intended for use between a web server and browser. Railgun is back-end for our network and automatically generates dictionaries.

http://calendar.perfplanet.com/2012/efficiently-compressing-...

Re: Bandwidth needs halved by new compression written in Go

#12

Post by abcd_f »

jgrahamc wrote: post #5
0x0 wrote: post #4

Earlier quoted context omitted.

It sounds like they reinvented rsync to me?

No, because we have more information than rsync does. We own both ends of the connection and can keep versions synchronized.

> we have more information than rsync does.

To what end? Rsync too works off both copies.

Re: Bandwidth needs halved by new compression written in Go

#13

Post by bitcartel »

The bandwidth reduction is due to use of a binary protocol, not Go. It just so happens the server code is written in Go and C.

From the article:

“Go is very light,” he said, “and it has fundamental support for concurrent programming. And it’s surprisingly stable for a young language. The experience has been extremely good—there have been no problems with deadlocks or pointer exceptions.” But the code hit a bit of a performance bottleneck under CloudFlare’s heavy loads, particularly because of its cryptographic modules—all of Railgun’s traffic is encrypted from end to end. “We swapped some things out into C just from a performance perspective," Graham-Cumming said.

“We want Go to be as fast as C for these things,” he explained, and in the long term he believes Go’s cryptographic modules will mature and get better. But in the meantime, “we swapped out Go’s native crypto for OpenSSL,” he said, using assembly language versions of the C libraries.

Re: Bandwidth needs halved by new compression written in Go

#14

Post by corresation »

jgrahamc wrote: post #11
corresation wrote: post #6

I was just looking into what SDCH is (an accept-encoding option from Chrome) and it sounds very, very similar: It generates a dictionary and then uses VCDIFF between requests. Is this related somehow?

Vaguely. Both Railgun and SDHC work by compressing web pages against an external dictionary. In SDHC the dictionary must be generated (somehow), and it is intended for use between a web server and browser. Railgun is back-end for our network and automatically generates dictionaries. http://calendar.perfplanet.com/2012/efficiently-compressing-...

That is superbly illuminating. Thank you.

Re: Bandwidth needs halved by new compression written in Go

#15

Post by jgrahamc »

0x0 wrote: post #9
jgrahamc wrote: post #5

Earlier quoted context omitted.

No, because we have more information than rsync does. We own both ends of the connection and can keep versions synchronized.

That sounds interesting, could you elaborate on how it is different from rsync though? "Keep versions synchronized" is a bit vague

The piece in the CloudFlare network and the piece in the customer network are able to keep track of which page versions they each have and so the part in the CloudFlare network sends a request saying "Please do GET /foo and compress it against version X". That means that at request time there's no back-and-forth between the components deciding what compression dictionary to use.

Re: Bandwidth needs halved by new compression written in Go

#16

Post by jgrahamc »

bitcartel wrote: post #13

The bandwidth reduction is due to use of a binary protocol, not Go. It just so happens the server code is written in Go and C. From the article: “Go is very light,” he said, “and it has fundamental support for concurrent programming. And it’s surprisingly stable for a young language. The experience has been extremely good—there have been no problems with deadlocks or pointer exceptions.” But the code hit a bit of a p…

The binary protocol means we don't add (much) overhead, the bandwidth reduction is because we are sending page diffs which themselves are encoded in a compact binary format.

Re: Bandwidth needs halved by new compression written in Go

#17

Post by 0x0 »

jgrahamc wrote: post #15
0x0 wrote: post #9

Earlier quoted context omitted.

That sounds interesting, could you elaborate on how it is different from rsync though? "Keep versions synchronized" is a bit vague

The piece in the CloudFlare network and the piece in the customer network are able to keep track of which page versions they each have and so the part in the CloudFlare network sends a request saying "Please do GET /foo and compress it against version X". That means that at request time there's no back-and-forth between the components deciding what compression dictionary to use.

A bit like rsync's --fuzzy or --compare-dest then?

Re: Bandwidth needs halved by new compression written in Go

#18

Post by jgrahamc »

0x0 wrote: post #17
jgrahamc wrote: post #15

Earlier quoted context omitted.

The piece in the CloudFlare network and the piece in the customer network are able to keep track of which page versions they each have and so the part in the CloudFlare network sends a request saying "Please do GET /foo and compress it against version X". That means that at request time there's no back-and-forth between the components deciding what compression dictionary to use.

A bit like rsync's --fuzzy or --compare-dest then?

Well, fuzzy tries to find something to use as a 'destination' file so it can send across some hashes. Railgun has more complete information because it is keeping synchronized and thus the part making a request can specify the dictionary to compress with in a single hash.

Re: Bandwidth needs halved by new compression written in Go

#19

Post by DannyBee »

abcd_f wrote: post #12
jgrahamc wrote: post #5

Earlier quoted context omitted.

No, because we have more information than rsync does. We own both ends of the connection and can keep versions synchronized.

> we have more information than rsync does. To what end? Rsync too works off both copies.

rsync is going to perform checksums on blocks to see if the blocks are the same. It transmits these checksums, and where the checksums differ, it deltas the blocks. Note that insertion/deletion in a file can push block boundaries off between two files, causing a problem known as "stream alignment", which can cause your binary delta to be much larger because it doesn't realize the block really shifted 16384 bytes over (or whatever), and so it thinks the client really doesn't have any of the bytes of that block.

In any case, if you know the files are related, you

1. Don't need to do any of this. You can simply send the binary delta that is is usually copy/add instructions (IE copy offset 16384, length 500 to offset 32768)

2. Can precompute the deltas.

You can actually precompute in any case, it just makes no sense unless you know you will be diffed against something else.

Re: Bandwidth needs halved by new compression written in Go

#20

Post by 0x0 »

jgrahamc wrote: post #18
0x0 wrote: post #17

Earlier quoted context omitted.

A bit like rsync's --fuzzy or --compare-dest then?

Well, fuzzy tries to find something to use as a 'destination' file so it can send across some hashes. Railgun has more complete information because it is keeping synchronized and thus the part making a request can specify the dictionary to compress with in a single hash.

Thanks for the explanation, that does sound useful! :)
Post reply on HN