Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preventing server overload: limit requests being processed

evanjones.ca

Post reply on HN
Print view View original

41–50 of 69 posts

Re: Preventing server overload: limit requests being processed

#41

Post by EGreg »

My recommendations for rate limiting:

1) Sign session ids that you issue and reject requests that don't have a valid signature. This can be done entirely in software at the router level without any I/O

2) For each session, do authentication. Unauthenticated sessions get lower caps.

3) For authenticated sessions, have various schemes for X units of Y in Z seconds. And then use this database - sharded by Y - before each expensive Y.

4) Y should be prefixed with a user role or payment plan or whatever, followed by the actual resource id. So people can buy another payment plan.

5) Possibly let people pay to access a resource beyond the quota.

6) Make clients recognize quota errors and retry with exponential backoff.

7) Host static resources in the app, fallback to a CDN, and only host dynamic resources on your own source servers.

8) Wherever possible, cache things in the client, keeping in mind that you have to evict the lease recently used items eventually.

Re: Preventing server overload: limit requests being processed

#42

Post by zeta0134 »

wutbrodo wrote: post #37

> Over Quota This application is temporarily over its serving quota. Please try again later. I'm actually not super sure is this is a single-serving joke site like isthemissiononfire.com, or if this is just a hilarious coincidence.

The Google Cache for it seems to be good, for anyone still getting this error. (It's still down for me.)

http://webcache.googleusercontent.com/search?q=cache:1oBS2v-...

Re: Preventing server overload: limit requests being processed

#43

Post by EGreg »

Animats wrote: post #35

I've used fair queuing to solve this problem. I have a site and API which rates other sites, checking them for ad links and trying to match them to real world business records. Inspecting a site takes 10 seconds to 2 minutes. The API, which is usually used from browser add-ons which tag web search results, lets anyone request the rating info for a site. Most requests are already cached and return immediately. Request…

What does the rating consist of, and what is its purpose?

Re: Preventing server overload: limit requests being processed

#44

Post by jonhohle »

kyrra wrote: post #5

[I'm a Google employee, opinions are my own] The GCP team did a blog post similar to this about using Load Shedding to survive a spike[0]. It's definitely a great way to survive a sudden spike in traffic, but not optimal, as serving errors is also bad (unless your clients have some sort of retry mechanism in-place). But if your clients do have a retry mechanism in place (especially one that has a backoff time on retr…

Retries aren't the only option if you control (or can at least influence) the client. Fallback sources and sane (or customer preferential) defaults are some additional options when the primary source of truth is unavailable. Both of these bypass thundering heard situations where recovery can be difficult even if the application returns to full health.

As a concrete example, when I worked at a large online retailer, we had to check certain customer properties during checkout. In the optimal case, we went to a service which fronted a consistent, transactional database. If that service failed or timed out, we had the option to hit a fast cache which generally was only a few milliseconds out of date. Finally, we could use less authoritative data that came from the application earlier, and if present, would always be in the customer's favor. That made it unlikely that our service ever appeared "down" regardless of the state of the dependencies.

Re: Preventing server overload: limit requests being processed

#45

Post by usaphp »

kyrra wrote: post #5

[I'm a Google employee, opinions are my own] The GCP team did a blog post similar to this about using Load Shedding to survive a spike[0]. It's definitely a great way to survive a sudden spike in traffic, but not optimal, as serving errors is also bad (unless your clients have some sort of retry mechanism in-place). But if your clients do have a retry mechanism in place (especially one that has a backoff time on retr…

Are you obligated to write that top part that the opinion is your own? What is the point of it? Just wondering.

Re: Preventing server overload: limit requests being processed

#46

Post by spraak »

STRML wrote: post #25

There was a native NodeJS module called "toobusy" that I rewrote a few years back to remove the native dependencies: https://www.npmjs.com/package/toobusy-js It's worth checking out - and helps you achieve this goal without necessarily having to come up with a req/s magicnum; instead you measure actual event loop latency to detect a slowdown.

That's really neat to actually measure event loop latency

Re: Preventing server overload: limit requests being processed

#48

Post by kyrra »

usaphp wrote: post #45
kyrra wrote: post #5

[I'm a Google employee, opinions are my own] The GCP team did a blog post similar to this about using Load Shedding to survive a spike[0]. It's definitely a great way to survive a sudden spike in traffic, but not optimal, as serving errors is also bad (unless your clients have some sort of retry mechanism in-place). But if your clients do have a retry mechanism in place (especially one that has a backoff time on retr…

Are you obligated to write that top part that the opinion is your own? What is the point of it? Just wondering.

Any time I'm posting something that may endorse "the company" I tend to put that as a clear marker that I am likely biased in what I'm saying.

Also, most large companies have something in the employment contract that talks about what you are allowed to post online (especially when it relates to the company).

Re: Preventing server overload: limit requests being processed

#49

Post by ot »

evanj wrote: post #28
ot wrote: post #15

A classical solution is to use a control algorithm (for example PI or PID) to avoid having to estimate the overload threshold: the server will just start shedding a fraction of the requests to maintain some metric (CPU utilization, latency, queue length, ...) below some target. See for example [1]. This is about background jobs, but the same principles apply to load shedding. [1] http://folk.ntnu.no/skoge/prost/proce…

Thanks for the link, the abstract seems relevant. I agree: a control algorithm seems like it should work. The challenge is figuring out the metrics and parameters to make something that works "well enough" for most applications, like TCP does for networks. I would really love for someone to figure that out, so we don't run into this very often.

Yeah, it's something that needs to be tuned per-service. It is quite easy if the service is X-bound for some local resource X (CPU, disk, flash, network card), but if it is bottlenecked on external service calls it can be quite challenging to define a representative stress metric.

Re: Preventing server overload: limit requests being processed

#50

Post by thunderbong »

Animats wrote: post #35

I've used fair queuing to solve this problem. I have a site and API which rates other sites, checking them for ad links and trying to match them to real world business records. Inspecting a site takes 10 seconds to 2 minutes. The API, which is usually used from browser add-ons which tag web search results, lets anyone request the rating info for a site. Most requests are already cached and return immediately. Request…

That's really interesting. Can you tell more about how it is implemented technology wise?
Post reply on HN