Caching the uncacheable: CloudFlare's Railgun

This looks like an interesting option for static content. But what still is not cachable of course is every request that has to hit a database (or at least clear parts of a cache).

In my experience (e.g. a site with 50.000-100.000 pageviews/day) it's mostly the requests hitting the db and not server throughput that are the bottleneck.

So for some people this could solve the wrong problem...

For some values of "huge". I was using 1Gbps as the link size, since that's almost certainly the uplink on the server, and thus an upper bound on the smallest link.

It would possibly be fair to use something closer to 155Mbps in a lot of places (the most constrained part of the link; we're not even talking about congestion/packet loss, which would exponentially favor this technique, and which does happen on SP transit links sometimes). At that point, 4MB could actually matter:

155Mbps = 20 MB/sec. 4MB takes 200ms to transfer; 100KB takes 5ms. If you assume a single packet for the delta instead, I'd be happy to save 5-200ms.

That's 5-200ms from the upstream server to the downstream cache, in the event the content needs up to be fetched. This would only be affect requests for new content.

If it'd be from upstream server (or cache) to end clients, that'd be an awesome saving.

Assume for the sake of argument that processing is free and instantaneous, and that the cache is in-line with the normal network path (and much nearer to the end user than the content server). This is arbitrarily close to true as Cloudflare expands.

So you're actually saving these 5-200ms vs. non-cloudflare delivery. It might still be 305-505ms total page load time, or easily greater (I used GEO sat for a while, feeding cell; it was hell), but if it were 305 instead of 505 I'd be quite happy actually.

(I'm assuming a 4MB page with 100KB which actually changes between loads. A 4MB all dynamic page, frequently reloaded, which can't be cached but where a tiny delta is possible would be pretty pathological.)

With CDNs you can also get benefits from prefetch/prefill, either opportunistically (due to multiple users hitting the same thing, or through actual scheduled fill). Works for many media assets but not for dynamically generated pages, which is what Railgun is supposed to address.

Exec summary: This is not caching. It is reduced latency for your visitors by removing connection setup over a large geographical distance. It also reduces the latency your web server has to deal with to effectively zero, but you can do that with Nginx. It also will reduce your bandwidth consumption from your providers perspective to a fraction of what it was.

I think describing this as caching hides the real benefits. My startup pushes 150 Mbps on average so I care about this stuff.

Rather than being caching, this introduces a proxy server that is geographically close to your visitors that then communicates with your server using an efficient compression protocol. So, much of the network path that would transfer a full payload is replaced by a compressed connection that does not have to build up and tear down a TCP connection with a three way handshake. The most important benefit I see here for site visitors is reducing latency by removing the connection setup. They will spend a few ms setting up the connection with a server a few hundred miles from them instead of on the other side of the planet. That server then serves a cached page or uses an established connection to only get the changes, which could mean as little as a single send and single receive packet.

Another benefit of this is that your local web server will be talking to a local cloudflare client which means there is practically zero latency from your perspective for each request. This means that each of your app server instances spends less time waiting for it's client to send or receive data and more time serving app requests. It's why people put Nginx in front of Apache.

I think the most important cost benefit here is reducing your bandwidth consumption. We're constantly negotiating our colo deal based on 95th percentile and getting your throughput from 1 Gbps down to 50 Mbps which I think this may do will drastically reduce your real hosting costs. Of course Cloudflare need to maintain their servers and will be serving 1Gbps to your customers but those cloudflare servers will be geographically closer to your customers. However because data centers bill based on your throughput at the switch and not how far your customers are away from you, I don't see that there are any cost savings they (cloudflare) can pass on to you. They're going to be billed what you were being billed for bandwidth, but they'll mark it up. I suppose you could argue there are economies of scale they benefit from, but that doesn't seem like a compelling argument for reduced costs.