Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bandwidth needs halved by new compression written in Go

arstechnica.com

Post reply on HN
Print view View original

101–110 of 121 posts

Re: Bandwidth needs halved by new compression written in Go

#101

Post by eru »

shanelja wrote: post #75
pivo wrote: post #69

Earlier quoted context omitted.

I'm curious, if you can get things done more quickly in Go, why do you feel C++ is generally a better choice? Performance?

Performance, I also feel more in control, it's hard to describe the feeling, it's just as though Go is providing an abstracted interface to the hardware, where as C++ provides raw, unfiltered but potentially dangerous access. Perhaps it's just my personal experience though.

> where as C++ provides raw, unfiltered but potentially dangerous access.

Wouldn't that be assembly? Last time I heard Stroustrup, he was all raving about abstractions, not raw unfiltered access.

Re: Bandwidth needs halved by new compression written in Go

#102

Post by coldtea »

0x0 wrote: post #4
calinet6 wrote: post #2

Go is used, sure, but the cool part about this is the binary Railgun protocol. Really smart. Send only file hashes and binary diffs back and forth, do a little extra computation to figure out the changes, but only send the absolute minimum data you need to the CDN. That's just smart, and frankly, I hope other CDNs have been doing this already, because at any high volume it seems to be an obvious solution. So that bri…

It sounds like they reinvented rsync to me?

Because every diff algorithm "reinvents rsync", right?

Re: Bandwidth needs halved by new compression written in Go

#103

Post by packetslave »

corresation wrote: post #81
emmelaich wrote: post #77

Earlier quoted context omitted.

I was going to mention Riverbed, glad someone else did. They've saved a huge amount for us (I think of 90%) of AJP (http tomcat) traffic. Not particularly difficult to set up.

Riverbed looks ideal but aren't they incredibly expensive? We looked at it years ago and I believe the necessary endpoints in our data center and in Singapore pushed past $140,000.

Riverbed (and all of the players in this space, really) are quite expensive, but this is where you get into the whole "Total ROI" argument for justifying the purchase.

Most companies depreciate hardware over 3 years. How much WAN/Internet bandwidth will you NOT use over the next 3 years, and how does that translate into upgrades you won't need to make?

There are also arguments for these boxes along the lines of "right now we use really expensive WAN links, but these boxes do end-to-end encryption too, so we can put the traffic on the Internet instead" but that opens up a few obvious cans of worms (and can of course be done without an accelerator with VPNs and whatnot).

Then you get into the more nebulous arguments that big bosses tend to like, such as "The average user makes Y XML requests per day to process X widgets. Each request takes Q seconds now. If we lower that to Q*0.5 with WAN acceleration, each user can now process N more widgets per day". Fluffy argument, but can have a big impact on business decision makers, especially if you can tie it to a dollar amount.

Note that WAN Accelerator salespeople are really, really good at coming up with arguments like this for/with you during the sales process.

Re: Bandwidth needs halved by new compression written in Go

#105

Post by zobzu »

huhtenberg wrote: post #28
DannyBee wrote: post #19

Earlier quoted context omitted.

rsync is going to perform checksums on blocks to see if the blocks are the same. It transmits these checksums, and where the checksums differ, it deltas the blocks. Note that insertion/deletion in a file can push block boundaries off between two files, causing a problem known as "stream alignment", which can cause your binary delta to be much larger because it doesn't realize the block really shifted 16384 bytes over…

I always thought rsync detected block moves and that's what made it a worthy PhD thesis.

I thought that too. It'd be interesting to see a comparison of the two software designs with actual difference in resource usage (cpu, io, bandwidth).

That would be really cool in fact.

Re: Bandwidth needs halved by new compression written in Go

#106

Post by radd9er »

zobzu wrote: post #104

Uho. Binary protocol. The problem being, it's actually bringing financial advantages over HTTP. HTTP has the advantage of being standard, simple, plain text and thus easy to work with. Hopefully http2.0 will attempt solving this.. erm...

is your concern that the proprietary protocols will take over the web?

Re: Bandwidth needs halved by new compression written in Go

#107

Post by radd9er »

philiac wrote: post #91

The article mentions how this compression technique is similar to image compression. Would anyone care to explain, in detail if necessary, how this is so? Thanks.

I think its because a whole bitmap isnt streamed for every new frame, just a diff telling the player about the parts of the map that need updating.

Re: Bandwidth needs halved by new compression written in Go

#108

Post by 23david »

shanelja wrote: post #32
jgrahamc wrote: post #24

Earlier quoted context omitted.

I agree. The benefit of using Go is that it's fast to write and has good concurrency features. To give you an idea of the size, there are 7,329 lines of Go code in Railgun (including comments) and a 6,602 line test suite. In the process we've committed various things back to Go itself and at some point I'll write a blog on the whole experience, but one thing that made a big difference was to write a memory recycler s…

http://www.jobscore.com/jobs/cloudflare/technical-customer-s... Not going to lie - I'm heavily considering taking this as an entry level position to get my foot in the door.

How about just emailing them? I feel like I've seen these kinds of 'wow I where do I apply for a job' posts on cloudflare news articles before. Smells like astroturfing.

Re: Bandwidth needs halved by new compression written in Go

#109

Post by zurn »

gatherknwldg wrote: post #76
jgrahamc wrote: post #24

Earlier quoted context omitted.

I agree. The benefit of using Go is that it's fast to write and has good concurrency features. To give you an idea of the size, there are 7,329 lines of Go code in Railgun (including comments) and a 6,602 line test suite. In the process we've committed various things back to Go itself and at some point I'll write a blog on the whole experience, but one thing that made a big difference was to write a memory recycler s…

"write a memory recycler" sigh This is by far go's biggest wart IMO, and one that frequently sends me back to a pauseless (hah! at least less pausy:) systems language. I sure do like it in almost every other meaningful regard. But I wish latency wasn't something the designers punted on.

> I wish latency wasn't something the designers punted on.

The simplistic GC isn't part of the language design, it's a stopgap in the first version.

Re: Bandwidth needs halved by new compression written in Go

#110

Post by pjmlp »

shanelja wrote: post #75
pivo wrote: post #69

Earlier quoted context omitted.

I'm curious, if you can get things done more quickly in Go, why do you feel C++ is generally a better choice? Performance?

Performance, I also feel more in control, it's hard to describe the feeling, it's just as though Go is providing an abstracted interface to the hardware, where as C++ provides raw, unfiltered but potentially dangerous access. Perhaps it's just my personal experience though.

> Performance, I also feel more in control, it's hard to describe the feeling, it's just as though Go is providing an abstracted interface to the hardware, where as C++ provides raw, unfiltered but potentially dangerous access.

Funny I though C++ did exactly the same thing. Where are the L1, L2 and L3 caches references, multiple opcode execution pipelines, processor instructions ?

Post reply on HN