The change detection algorithm is clever. But this is a classic memory vs. processor problem. The real trick here is that the Railgun service instantly adds massive amounts of cache to your service; it just so happens - if their claims aren't inflated - adding these additional resources to your service is transparent. This has nothing to do with Railgun being developed on Go.
Bandwidth needs halved by new compression written in Go
71–80 of 121 posts
Re: Bandwidth needs halved by new compression written in Go
#72Earlier quoted context omitted.
I agree. The benefit of using Go is that it's fast to write and has good concurrency features. To give you an idea of the size, there are 7,329 lines of Go code in Railgun (including comments) and a 6,602 line test suite. In the process we've committed various things back to Go itself and at some point I'll write a blog on the whole experience, but one thing that made a big difference was to write a memory recycler s…
Go does look awesome. I've spent some time with Erlang, Clojure and Scala (roughly in the order that I liked them most), but Go passed the "get started writing useful code quickly" test better than any of them. Haven't gone beyond the basics yet, but I think it might occupy a sweet spot of ease of use combined with "power", loosely defined.
Erlang with OTP also does pretty well on that test; I've deployed non-trivial programs that weighed in at a couple hundred lines of code.
I agree though — Go looks awesome.
Re: Bandwidth needs halved by new compression written in Go
#73Earlier quoted context omitted.
bsdiff is not a general purpose binary delta algorithm, it's targeted at executables. When you change a single line in the source code of a program and recompile it, bsdiff produces a small diff, even though a normal binary diff between the old and new executable would be huge due to how even a single extra instruction can cause many more addresses to shift. bsdiff wouldn't be particularly useful here.
This is true. Re-reading the bsdiff paper, it's pretty tailored to executable file formats. http://www.daemonology.net/papers/bsdiff.pdf
It works fine on non-executables too. Executables are the hard case, that's all.
Re: Bandwidth needs halved by new compression written in Go
#74Earlier quoted context omitted.
Ok, that sounds great! (Same to the comment above, replying to this one for continuity) - let me mull it over this week. And I couldn't agree more, being placed in the firing lines of customers is often more telling than building the software yourself - "normal" people tend to notice things which we as developers are prone to miss or gloss over unintentionally. ----- The awkward moment when I notice I blanked the CEO
Mull it over a week?
Large changes in lifestyle, such as a complete relocation, new job, etc, should not be something undertaken lightly - if I interviewed and got offered the job I would be under a lot of pressure, which I can mitigate now by thinking more carefully before undertaking anything, I wouldn't want to waste both my time and the time of the people at Cloudflare by making an important decision without carefully weighing the pros and cons.
Re: Bandwidth needs halved by new compression written in Go
#75Earlier quoted context omitted.
I've never used go professionally and most of my spare time is split between C++ and Scheme at the moment, but when I did go spelunking with Go, I found it a breeze to write complicated functionality in it - it felt like C++, but easier and more initially powerful. I still feel that C++ is generally a better choice, but if I only had a short time to write something in, I would definitely go for Go.
I'm curious, if you can get things done more quickly in Go, why do you feel C++ is generally a better choice? Performance?
Performance, I also feel more in control, it's hard to describe the feeling, it's just as though Go is providing an abstracted interface to the hardware, where as C++ provides raw, unfiltered but potentially dangerous access.
Perhaps it's just my personal experience though.
Re: Bandwidth needs halved by new compression written in Go
#76The title suggests that there's something unique about Go, either the language or its standard library, that enables bandwidth savings. In fact, Cloudflare have written some software which they claim enables them to reduce their bandwidth, and this software happens to be written in Go. This might be an excellent choice (and I suspect it probably is), but it's not Go per se that is reducing the bandwidth usage.
I agree. The benefit of using Go is that it's fast to write and has good concurrency features. To give you an idea of the size, there are 7,329 lines of Go code in Railgun (including comments) and a 6,602 line test suite. In the process we've committed various things back to Go itself and at some point I'll write a blog on the whole experience, but one thing that made a big difference was to write a memory recycler s…
"write a memory recycler"
sigh This is by far go's biggest wart IMO, and one that frequently sends me back to a pauseless (hah! at least less pausy:) systems language. I sure do like it in almost every other meaningful regard. But I wish latency wasn't something the designers punted on.
Re: Bandwidth needs halved by new compression written in Go
#77Earlier quoted context omitted.
What you describe is exactly what I've been looking for. There are remarkably few resources on this. We have users in Singapore who access various XML-heavy web services in our NY office. A dictionary-style over-multiple-requests compression technique would be brilliant for their case.
Take a look at the various WAN accelerator appliances (Cisco, Silverpeak, Riverbed). They do almost exactly what it sounds like you want (if I'm remembering back to my evaluations, Cisco at least uses a multi-request dictionary for their compression)
I was going to mention Riverbed, glad someone else did.
They've saved a huge amount for us (I think of 90%) of AJP (httptomcat) traffic. Not particularly difficult to set up.
Re: Bandwidth needs halved by new compression written in Go
#78Earlier quoted context omitted.
I'm curious, if you can get things done more quickly in Go, why do you feel C++ is generally a better choice? Performance?
Performance, I also feel more in control, it's hard to describe the feeling, it's just as though Go is providing an abstracted interface to the hardware, where as C++ provides raw, unfiltered but potentially dangerous access. Perhaps it's just my personal experience though.
That kind of fits Rob Pike's explanation for why Go isn't more popular with C++ programmers (although he gave it a negative spin).
In a neutral way, it's like: if you spent all that effort mastering this language to get such fine-grained control, why would you give that up again? And really, I understand: why would you give that up? Especially if you know how to use C++ in a fairly painless way.
Re: Bandwidth needs halved by new compression written in Go
#79Earlier quoted context omitted.
Performance, I also feel more in control, it's hard to describe the feeling, it's just as though Go is providing an abstracted interface to the hardware, where as C++ provides raw, unfiltered but potentially dangerous access. Perhaps it's just my personal experience though.
That kind of fits Rob Pike's explanation for why Go isn't more popular with C++ programmers (although he gave it a negative spin). In a neutral way, it's like: if you spent all that effort mastering this language to get such fine-grained control, why would you give that up again? And really, I understand: why would you give that up? Especially if you know how to use C++ in a fairly painless way.
Do you have a link to that? I would be interested to read it.
I also couldn't agree more, but normally, people retort with the Hammer and Screwdriver argument.
Re: Bandwidth needs halved by new compression written in Go
#80Earlier quoted context omitted.
The piece in the CloudFlare network and the piece in the customer network are able to keep track of which page versions they each have and so the part in the CloudFlare network sends a request saying "Please do GET /foo and compress it against version X". That means that at request time there's no back-and-forth between the components deciding what compression dictionary to use.
A bit like rsync's --fuzzy or --compare-dest then?
Don't you understand? We need you to accept that this is a new technology and a ground-breaking algorithm and a new innovative (and valuable, non-obvious) technology. CloudFlare was established in 2007 with the goal to develop a faster, safter, better internet. CloudFlare, the web performance and security company, set records this month hitting more than 100 million daily active users and more than 50 billion monthly page views!