Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bandwidth needs halved by new compression written in Go

arstechnica.com

Post reply on HN
Print view View original

41–50 of 121 posts

Re: Bandwidth needs halved by new compression written in Go

#41

Post by newman314 »

songgao wrote: post #23
corresation wrote: post #6

I was just looking into what SDCH is (an accept-encoding option from Chrome) and it sounds very, very similar: It generates a dictionary and then uses VCDIFF between requests. Is this related somehow?

Railgun is used between CDN and http server, while this one seems to be between browser and http server. Railgun only requires website to deploy a client, and cooperate with cloudflare. User's client doesn't have to be Chrome or whatever; web server doesn't need to be aware of Railgun. It's transparent to both HTTP clients and HTTP servers. SDCH, however, requires a modification to HTTP/1.1 protocol, which implies ch…

Until apache or nginx implements it, I wouldn't really see SDCH as gaining any real traction.

AFAIK, Chrome is the only one supporting it from a browser standpoint.

Re: Bandwidth needs halved by new compression written in Go

#42

Post by jgrahamc »

corresation wrote: post #14
jgrahamc wrote: post #11

Earlier quoted context omitted.

Vaguely. Both Railgun and SDHC work by compressing web pages against an external dictionary. In SDHC the dictionary must be generated (somehow), and it is intended for use between a web server and browser. Railgun is back-end for our network and automatically generates dictionaries. http://calendar.perfplanet.com/2012/efficiently-compressing-...

That is superbly illuminating. Thank you.

It seems like SDCH has been around for 4 years, I presume the lack of data means it hasn't worked out.

A barrier to implementation of SDCH is deciding what dictionaries to create and when to update them.

Re: Bandwidth needs halved by new compression written in Go

#43

Post by corresation »

jws wrote: post #40
jgrahamc wrote: post #11

Earlier quoted context omitted.

Vaguely. Both Railgun and SDHC work by compressing web pages against an external dictionary. In SDHC the dictionary must be generated (somehow), and it is intended for use between a web server and browser. Railgun is back-end for our network and automatically generates dictionaries. http://calendar.perfplanet.com/2012/efficiently-compressing-...

Is anyone aware of a performance analysis between SDCH and one of the dynamic compressions like deflate? I google, but all I find is people complaining their proxy/filter/appliance/diagnostic is breaking because it doesn't understand SDCH. It seems like SDCH has been around for 4 years, I presume the lack of data means it hasn't worked out. (I imagine that you could drastically reduce the CPU load of compression by m…

What you describe is exactly what I've been looking for. There are remarkably few resources on this.

We have users in Singapore who access various XML-heavy web services in our NY office. A dictionary-style over-multiple-requests compression technique would be brilliant for their case.

Re: Bandwidth needs halved by new compression written in Go

#44

Post by xxdesmus »

shanelja wrote: post #39
xxdesmus wrote: post #38

Earlier quoted context omitted.

Please do consider applying if you're interested. We are actively looking for qualified technical folks.

I live in Lancashire at the moment, the only issue for me would really be spending £150 on a round trip to London, do you guys do preliminary Skype interviews? (I'm aware that the post might not be prestigious as say, engineering - however, I feel that having someone with strong web development experience (who is a user of Cloudflare already) would more than offset the slight inconvenience on your part.) EDIT: Gramma…

We definitely do the initial interview via phone/Skype. There absolutely is room to grow and move into other areas of the company with experience. I highly recommend becoming familiar with the platform through the "front lines" of support. It gives engineers a different perspective on our service.

Re: Bandwidth needs halved by new compression written in Go

#45

Post by eastdakota »

shanelja wrote: post #39
xxdesmus wrote: post #38

Earlier quoted context omitted.

Please do consider applying if you're interested. We are actively looking for qualified technical folks.

I live in Lancashire at the moment, the only issue for me would really be spending £150 on a round trip to London, do you guys do preliminary Skype interviews? (I'm aware that the post might not be prestigious as say, engineering - however, I feel that having someone with strong web development experience (who is a user of Cloudflare already) would more than offset the slight inconvenience on your part.) EDIT: Gramma…

We typically conduct the first interviews on the phone or Skype for interesting candidates. If it makes sense to do in-person interviews, we're happy to cover the cost of transportation for candidates we're excited about. In other words, if you're excited about working with CloudFlare, don't let the £150 stand in the way of applying.

Re: Bandwidth needs halved by new compression written in Go

#46

Post by peterwwillis »

DoubleCluster wrote: post #26

This is WAN optimization, right? This is already being done but usually for (VPN) connections to other branches of a company.

No. This is basically binary diffing and compression.

Edit: err, you are correct, I didn't realize WAN optimization included binary diffing and compression. Should google before I comment.

Re: Bandwidth needs halved by new compression written in Go

#47

Post by shanelja »

xxdesmus wrote: post #44
shanelja wrote: post #39

Earlier quoted context omitted.

I live in Lancashire at the moment, the only issue for me would really be spending £150 on a round trip to London, do you guys do preliminary Skype interviews? (I'm aware that the post might not be prestigious as say, engineering - however, I feel that having someone with strong web development experience (who is a user of Cloudflare already) would more than offset the slight inconvenience on your part.) EDIT: Gramma…

We definitely do the initial interview via phone/Skype. There absolutely is room to grow and move into other areas of the company with experience. I highly recommend becoming familiar with the platform through the "front lines" of support. It gives engineers a different perspective on our service.

Ok, that sounds great! (Same to the comment above, replying to this one for continuity) - let me mull it over this week.

And I couldn't agree more, being placed in the firing lines of customers is often more telling than building the software yourself - "normal" people tend to notice things which we as developers are prone to miss or gloss over unintentionally.

-----

The awkward moment when I notice I blanked the CEO

Re: Bandwidth needs halved by new compression written in Go

#48

Post by justinsb »

I think this is just RFC 3229, with a binary protocol (?) http://www.ietf.org/rfc/rfc3229.txt

I've always thought there were some potential attacks there around cache disclosure (which Google avoided by going with SDCH instead).

CloudFlare controls the server and the client, so they don't need to worry about the attacks or about persuading everyone to adopt their RFC.

Re: Bandwidth needs halved by new compression written in Go

#49

Post by jgrahamc »

huhtenberg wrote: post #29
jgrahamc wrote: post #15

Earlier quoted context omitted.

The piece in the CloudFlare network and the piece in the customer network are able to keep track of which page versions they each have and so the part in the CloudFlare network sends a request saying "Please do GET /foo and compress it against version X". That means that at request time there's no back-and-forth between the components deciding what compression dictionary to use.

For what it's worth, this is fairly standard binary patching approach as used in software updates. I am aware of at least two mainstream titles that do this, and I'd be surprised if Firefox, for example, doesn't push updates this way. (edit) That's an awesome name by the way. Railgun.

Can't claim credit for the name. I wanted to call it Rocket Sled.

Re: Bandwidth needs halved by new compression written in Go

#50

Post by eastdakota »

jgrahamc wrote: post #49
huhtenberg wrote: post #29

Earlier quoted context omitted.

For what it's worth, this is fairly standard binary patching approach as used in software updates. I am aware of at least two mainstream titles that do this, and I'd be surprised if Firefox, for example, doesn't push updates this way. (edit) That's an awesome name by the way. Railgun.

Can't claim credit for the name. I wanted to call it Rocket Sled.

I'm glad we didn't call it Rocket Sled.
Post reply on HN