Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bandwidth needs halved by new compression written in Go

arstechnica.com

Post reply on HN
Print view View original

61–70 of 121 posts

Re: Bandwidth needs halved by new compression written in Go

#61

Post by sigil »

j_s wrote: post #57
sigil wrote: post #56

Question for jgrahamc: how much more efficient is your binary delta algorithm than cperciva's bsdiff [1]? I assume since you've got the preimages of compression, as well as control over the compression format, that the diff and patch operations are much more efficient in space and time than they would be with arbitrary binary data. But...by how much? [1] http://www.daemonology.net/bsdiff/

I am particularly interested in this aspect of the discussion (explaining the process leading to deciding to develop a new tech in-house instead of re-using any existing approach). In an ideal world there would be plenty of experimentation with real-world data to justify things, but I don't read about that happening too often.

Agree with you there on "profile first." But knowing jgrahamc, he did -- and I'd love to know the results.

Re: Bandwidth needs halved by new compression written in Go

#62

Post by lclarkmichalek »

shanelja wrote: post #30
bitcartel wrote: post #13

The bandwidth reduction is due to use of a binary protocol, not Go. It just so happens the server code is written in Go and C. From the article: “Go is very light,” he said, “and it has fundamental support for concurrent programming. And it’s surprisingly stable for a young language. The experience has been extremely good—there have been no problems with deadlocks or pointer exceptions.” But the code hit a bit of a p…

On another note, it's always nice to see such an influential part of the HN community giving quotes for sites like this - not only does it make me a little proud to be associated with any of you, it makes me more hopeful for the chances of my future that I can call myself one of us .

Success by association seems about as valid as guilt by association.

Re: Bandwidth needs halved by new compression written in Go

#64

Post by meh02 »

peterwwillis wrote: post #34
silvertonia wrote: post #7

Could be very cool. I couldn't get through the article because it read like a press release. Maybe if someone who hasn't been spoon-fed the story reports on it, I'll take notice.

I don't know why you're being downvoted, the article is written pretty shittily. The article is mostly just quotes from jgc and the CEO and some filler by the writer. Also the assertion that "It has already cut the bandwidth used by 4Chan and Imgur by half" sounds disingenuous and possibly not backed up by moot's quote “We've seen a ~50% reduction in backend transfer for our HTML pages (transfer between our servers a…

Considering that imgur serves... images... no, their transfer isn't being cut by 50%.

This article is just a press release about a feature that has been done forever (look up "WAN accelerator").

Re: Bandwidth needs halved by new compression written in Go

#65

Post by halcyondaze »

shanelja wrote: post #47
xxdesmus wrote: post #44

Earlier quoted context omitted.

We definitely do the initial interview via phone/Skype. There absolutely is room to grow and move into other areas of the company with experience. I highly recommend becoming familiar with the platform through the "front lines" of support. It gives engineers a different perspective on our service.

Ok, that sounds great! (Same to the comment above, replying to this one for continuity) - let me mull it over this week. And I couldn't agree more, being placed in the firing lines of customers is often more telling than building the software yourself - "normal" people tend to notice things which we as developers are prone to miss or gloss over unintentionally. ----- The awkward moment when I notice I blanked the CEO

Mull it over a week?

Re: Bandwidth needs halved by new compression written in Go

#66

Post by jgrahamc »

sigil wrote: post #61
j_s wrote: post #57

Earlier quoted context omitted.

I am particularly interested in this aspect of the discussion (explaining the process leading to deciding to develop a new tech in-house instead of re-using any existing approach). In an ideal world there would be plenty of experimentation with real-world data to justify things, but I don't read about that happening too often.

Agree with you there on "profile first." But knowing jgrahamc, he did -- and I'd love to know the results.

:-) Yes, that's very true.

Initially, I wasn't actually planning to do deltas for the compression technique and it was in testing with a whole bunch of common sites that I stumbled upon the fact that they don't change very much. That lead me to wonder about the algorithms that might be used.

I did test quite a lot of stuff (and at one point thought I'd come up with a truly cool new algorithm only to realize that I was mistaken :-) to decide what to do.

Railgun has to trade off three things: compression efficiency, space and time. Because we are trying to do this for performance time is the most important thing to optimize for, followed by efficiency, followed by space. bsdiff is very, very good at delta compressing binary things; Railgun isn't as good, but it's very, very fast.

Re: Bandwidth needs halved by new compression written in Go

#67

Post by jakubw »

sigil wrote: post #56

Question for jgrahamc: how much more efficient is your binary delta algorithm than cperciva's bsdiff [1]? I assume since you've got the preimages of compression, as well as control over the compression format, that the diff and patch operations are much more efficient in space and time than they would be with arbitrary binary data. But...by how much? [1] http://www.daemonology.net/bsdiff/

bsdiff is not a general purpose binary delta algorithm, it's targeted at executables. When you change a single line in the source code of a program and recompile it, bsdiff produces a small diff, even though a normal binary diff between the old and new executable would be huge due to how even a single extra instruction can cause many more addresses to shift. bsdiff wouldn't be particularly useful here.

Re: Bandwidth needs halved by new compression written in Go

#69

Post by pivo »

shanelja wrote: post #35
vanderZwan wrote: post #25

Earlier quoted context omitted.

I like Go, but that bothered me too. From my personal point of view I'm happy they used it though, because it means more people tinkering with and improving the language.

I've never used go professionally and most of my spare time is split between C++ and Scheme at the moment, but when I did go spelunking with Go, I found it a breeze to write complicated functionality in it - it felt like C++, but easier and more initially powerful. I still feel that C++ is generally a better choice, but if I only had a short time to write something in, I would definitely go for Go.

I'm curious, if you can get things done more quickly in Go, why do you feel C++ is generally a better choice? Performance?

Re: Bandwidth needs halved by new compression written in Go

#70

Post by sigil »

jakubw wrote: post #67
sigil wrote: post #56

Question for jgrahamc: how much more efficient is your binary delta algorithm than cperciva's bsdiff [1]? I assume since you've got the preimages of compression, as well as control over the compression format, that the diff and patch operations are much more efficient in space and time than they would be with arbitrary binary data. But...by how much? [1] http://www.daemonology.net/bsdiff/

bsdiff is not a general purpose binary delta algorithm, it's targeted at executables. When you change a single line in the source code of a program and recompile it, bsdiff produces a small diff, even though a normal binary diff between the old and new executable would be huge due to how even a single extra instruction can cause many more addresses to shift. bsdiff wouldn't be particularly useful here.

This is true. Re-reading the bsdiff paper, it's pretty tailored to executable file formats.

http://www.daemonology.net/papers/bsdiff.pdf

Post reply on HN