Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The 5-Hour CDN

fly.io

Post reply on HN
Print view View original

81–90 of 90 posts

Re: The 5-Hour CDN

#81

Post by skoskie »

arghwhat wrote: post #67
endymi0n wrote: post #56

Earlier quoted context omitted.

And to add the last big one from the trifecta: Know how to deal with cacheable data. Know how to deal with uncacheable data. But by all means, know how to keep them apart. Accidentally caching uncacheable data has lead so some of the most ugly and avoidable data leaks and compromises in recent times. If you go down the "route everything through a CDN route (that can be as easy as ticking a box in the Google Cloud Pla…

no-cache does not mean content must not be cached - in fact, it specifies the opposite! no-cache means that the response may be stored in any cache, but cached content MUST be revalidated before use. public means that the response may be cached in any cache even if the response was not normally cacheable, while private restricts this to only the user agent's cache. no-store specifies that this response must not be st…

That’s the best synopsis of the cache options I’ve ever read. It’s one of those things I have to pull documentation on every time I use it, but the way you just explained it makes so much sense that I might just memorize it now.

Edit: And now I see that you just copied bits from the Moz Dev page. I'll have to start using those more. I think the MS docs always come up first in Google.

Re: The 5-Hour CDN

#82

Post by dbbk »

simonw wrote: post #12

This article touches on "Request Coalescing" which is a super important concept - I've also seen this called "dog-pile prevention" in the past. Varnish has this built in - good to see it's easy to configure with NGINX too. One of my favourite caching proxy tricks is to run a cache with a very short timeout, but with dog-pile prevention baked in. This can be amazing for protecting against sudden unexpected traffic spi…

Is this the same idea as `stale-while-revalidate`?

Re: The 5-Hour CDN

#83

Post by arghwhat »

skoskie wrote: post #81
arghwhat wrote: post #67

Earlier quoted context omitted.

no-cache does not mean content must not be cached - in fact, it specifies the opposite! no-cache means that the response may be stored in any cache, but cached content MUST be revalidated before use. public means that the response may be cached in any cache even if the response was not normally cacheable, while private restricts this to only the user agent's cache. no-store specifies that this response must not be st…

That’s the best synopsis of the cache options I’ve ever read. It’s one of those things I have to pull documentation on every time I use it, but the way you just explained it makes so much sense that I might just memorize it now. Edit: And now I see that you just copied bits from the Moz Dev page. I'll have to start using those more. I think the MS docs always come up first in Google.

MDN docs are quite good at times. And yes, certain parts were copy pasted in, as I didn't want to accidentally end up spreading misinformation.

Also note that I only mentioned the usual suspects - there are many more options, like must-revalidate.

Re: The 5-Hour CDN

#84

Post by derefr »

thraxil wrote: post #63
8note wrote: post #41

Earlier quoted context omitted.

Hmm. I think of thundering Herd being about retries. All your failing requests batch up when your retry strategy sucks, then you end up really high traffic on every retry, and very little in between

Retries without jitter are indeed a common source of thundering herd problems. Even with exponential backoff, if all the clients are retrying simultaneously, they'll hammer your servers over and over. Adding jitter (just a random amount of extra delay that's different for every client+retry), they get staggered and the requests are spread out.

What do you do when you’re an API SaaS, and it’s your clients’ apps that are making thundering-herd requests?

Imagine you’re a service like Feedly, and one of your “direct customer” API clients — some feed-reader mobile client — has coded their apps such that all of their connected clients will re-request the specific user’s unique feed at exact, crontab-like 5-minute offsets from the start of the hour. So every five minutes, you get a huge burst of traffic, from all these clients—and it’s all different traffic, with nothing coalescesable.

You don’t control the client in this case, but nor can you simply ban them—they’re your paying customers! (Yes, you can “fire your customer”, but this would be most of your customers…)

And certainly, you can try to teach the devs of your client how to write their own jitter logic—but that rarely works out, as often it’s junior frontend devs who wrote the client-side code, and it’s hard to have a non-intermediated conversation with them.

Re: The 5-Hour CDN

#85

Post by thraxil »

derefr wrote: post #84
thraxil wrote: post #63

Earlier quoted context omitted.

Retries without jitter are indeed a common source of thundering herd problems. Even with exponential backoff, if all the clients are retrying simultaneously, they'll hammer your servers over and over. Adding jitter (just a random amount of extra delay that's different for every client+retry), they get staggered and the requests are spread out.

What do you do when you’re an API SaaS, and it’s your clients’ apps that are making thundering-herd requests? Imagine you’re a service like Feedly, and one of your “direct customer” API clients — some feed-reader mobile client — has coded their apps such that all of their connected clients will re-request the specific user’s unique feed at exact, crontab-like 5-minute offsets from the start of the hour. So every five…

If you have no control at all over the client, then ultimately, you have to just take it and build your service to handle that amount of traffic. Adding jitter is a technique that you use when writing clients. That's why I mentioned it in the context of retries. If you are writing a CDN per the article, at some point your CDN has to make requests back to the origin. If one of those requests fails and you retry, you add jitter there to avoid DoSing yourself. If you are working in a microservices architecture, you add jitter on retries between your services.

The best you can do with clients that are out of your control is to publish a client library/SDK for your API that is convenient for your customers to use and implements best practices like exponential backoff, jitter, etc. If you have documentation with code snippets that junior devs are likely to copy and paste, include it in those.

If you've painted yourself into a corner like you describe and are seeing extremely regular traffic patterns, you might be able to pre-cache. Ie, it's 12:01 and you know that a barrage is coming at 12:05. Start going down the list of clients/feeds that you know are likely to be requested based on recent traffic patterns and generate the response, putting it in your cache/CDN with a five minute TTL. Then at least a good portion of the requests should be served straight from there and not add load to the origin. There are obviously drawbacks/risks to that approach, but it might be all you can really do.

Re: The 5-Hour CDN

#86

Post by awoods187 »

mrkurt wrote: post #78
mbStavola wrote: post #64

Fly is great and I love reading their blog posts. Just hoping they come back around on CockroachDB-- I feel like it's a match made in heaven for what they're providing.

We're getting there: https://github.com/fly-apps/cockroachdb

PM at CRL here--we love Fly too! Definitely can see our two products working together!

Re: The 5-Hour CDN

#87

Post by lelandbatey »

derefr wrote: post #84
thraxil wrote: post #63

Earlier quoted context omitted.

Retries without jitter are indeed a common source of thundering herd problems. Even with exponential backoff, if all the clients are retrying simultaneously, they'll hammer your servers over and over. Adding jitter (just a random amount of extra delay that's different for every client+retry), they get staggered and the requests are spread out.

What do you do when you’re an API SaaS, and it’s your clients’ apps that are making thundering-herd requests? Imagine you’re a service like Feedly, and one of your “direct customer” API clients — some feed-reader mobile client — has coded their apps such that all of their connected clients will re-request the specific user’s unique feed at exact, crontab-like 5-minute offsets from the start of the hour. So every five…

If you're extremely desperate, you can start adding conditional jitter (somewhere within 5ms - 200 ms) to your load balancer/reverse proxy, such as your NGINX/Envoy/Apache box, which sits in front of your API. You can make the jitter conditional on count of concurrent requests or on latency spikes. It's an extreme last resort, and may require a bit of custom work via custom module or extension, but it is possible.

In general, try to avoid not having any control over the client and if you must lack control over the client (such as if you're a pure SaaS company selling a public API), you can apply jitter based on API key in addition to other metrics I mentioned above.

As better engineers than I used to say at a previous engagemen: "if it's not in the SLA, it's an opportunity for optimization"

Re: The 5-Hour CDN

#88

Post by derefr »

lelandbatey wrote: post #87
derefr wrote: post #84

Earlier quoted context omitted.

What do you do when you’re an API SaaS, and it’s your clients’ apps that are making thundering-herd requests? Imagine you’re a service like Feedly, and one of your “direct customer” API clients — some feed-reader mobile client — has coded their apps such that all of their connected clients will re-request the specific user’s unique feed at exact, crontab-like 5-minute offsets from the start of the hour. So every five…

If you're extremely desperate, you can start adding conditional jitter (somewhere within 5ms - 200 ms) to your load balancer/reverse proxy, such as your NGINX/Envoy/Apache box, which sits in front of your API. You can make the jitter conditional on count of concurrent requests or on latency spikes. It's an extreme last resort, and may require a bit of custom work via custom module or extension, but it is possible. In…

I like the “jitter based on API key” idea.

It’s somewhat hard in our case, as our direct customers (like the mobile app I mentioned) have API keys with us, but they don’t tell us about which user of theirs is making the request. And often they’ll run an HTTP gateway (in part so that they don’t have to embed their API key for our service in their client app), so we don’t even get to see the originating user IPs for these requests, either. We just get these huge spikes of periodic traffic, all from the same IP, all with the same API key, all about different things, and all delivered over a bunch of independent, concurrent TCP connections.

I’ve been considering a few options:

- Require users that have such a “multiple users behind an API gateway” setup, to tag their proxied requests with per-user API sub-keys, so we can jitter/schedule based on those.

- Since these customers like API gateways so much, we could just build a better API gateway for them to run; one that benefits us. (E.g. by Nagle-ing requests together into fewer, larger batch requests.) Requests that come as a single large batch request, could be scheduled by our backend at an optimal concurrency level, rather than trying to deal with huge concurrency bursts as we are now.

- Force users to rewrite their software to “play nice”, by introducing heavy-handed rate-limiting. Try to tune it so that the only possible way to avoid 429s is to either do gateway-side request queuing, or to introduce per-client schedule offsets (i.e. placing users on a hash ring by their ID, so for a periodic-per-5-minutes request, equal numbers of client apps are set to make the request at T+0, vs. T+2.5.)

- Introduce a middleware / reverse-proxy that holds an unbounded-in-size no-expire request queue, with one queue per API key, where requests are popped fairly from each queue (or prioritized according to the plan the user is paying for). Ensure backends only select(1) requests out from the middleware’s downstream sockets as quickly as they’re able to handle them. Require API requests to have explicit TTLs — a time after which serving the request would no longer be useful. If a backend pops a request and finds that it’s past its TTL, it discards it, answering it with an immediate 504 error.

Re: The 5-Hour CDN

#89

Post by hinkley »

spondyl wrote: post #13
cj wrote: post #8

Earlier quoted context omitted.

> If you’re okay with every request having the latency all the way to your origin, you can have the CDN revalidate its cache on every request. It's also worth mentioning that even when revalidating on every request (or not caching at all), routing through a CDN can still improve overall latency because the TLS can be terminated at a local origin server, significantly shortening the TLS handshake.

Ah, the TLS shortening aspect of a CDN is something that seems obvious in hindsight but I'd never really thought about it. Thanks!

Some of the HTTP/2 and HTTP3 design choices are seen as trying to solve this problem another way.

If a round trip to New York is too long, then twenty of them is way worse. So I can either do 20 round trips to Nevada, which does <20 round trips to Chicago, which does <<20 round trips to New York. Or, I can do some more cleverness with transport and session bootstrapping and end up with 14 round trips to New York.

Re: The 5-Hour CDN

#90

Post by philsnow »

samlambert wrote: post #80
philsnow wrote: post #54

Earlier quoted context omitted.

unrelated to CDNs but IIRC vitess did/does query coalescing too -- if it starts to serve a query for "select * from users where id = 123" and then another 20 connections all want the same query result, vitess doesn't send all 21 select queries to the backend, it sends the first one and then has all the connections wait on the backend response, then serves the same response to them all.

Vitess still does this. It can also do similar with writes on hot rows where someone is incrementing a counter for example.

Wait, how can it do it with writes/increments? Does it keep track of which rows are hot and add a short but stochastically distributed delay to writes to try to coalesce more updates into a single hit to the DB?

I would think you'd need to do it that way, you wouldn't want to reply "done" to the first increment if that operation is going to be batched up with other ops; you'd want to keep that connection hanging until all the increments you're going to aggregate have all been committed by the backend.

In the select coalescing case, except for bookkeeping overhead, none of the queries are slower (it's a big net win all around because not only do clients get their answers on average somewhat sooner, but the DB doesn't have to parse those queries, check for them in the query cache, or marshal N responses).

But in the increment/write case, it seems like in order to spare some DB resources, some clients will perceive increased write delays (or does it still net a win because the DB backend doesn't have to deal with the contention?).

Post reply on HN