Preventing server overload: limit requests being processed

> The Google SRE book has a chapter about server overload and cascading failures, which I highly recommend if you want to learn more.

This is basically just "stacks and queues" in a general sense, most often seen as wait-and-retry in an application, and rate controls in a service.

To do this and survive cascading failures you have to know the limits of each part of your stack and implement rate controls for each. An easy way to do this is to design network services that serve requests for applications via an API, rather than giving apps unfettered access to network resources. You can allow applications access to your network service and change rate controls for each app as needed. Alternately, you can add limits to your apps' layers at design time, rather than having to discover the limits by performance testing or trial-and-error.

The author wants a universal solution, which is a bit like saying, please solve "traffic" for me for every mode of transportation. Airplanes have different traffic than cars, so their designs will differ, although they all involve an agent that coordinates traffic flows according to a set of rules.

I have a site and API which rates other sites, checking them for ad links and trying to match them to real world business records. Inspecting a site takes 10 seconds to 2 minutes. The API, which is usually used from browser add-ons which tag web search results, lets anyone request the rating info for a site.

Most requests are already cached and return immediately. Requests for unknown sites are queued up for the rating engine, which runs in processes separate from the web-facing side, and return "wait" to the requestor. So there's a work queue to manage.

It's managed using fair queuing. If there is no pending request from the requesting IP address, the new request goes into the queue. If there's a request from the same IP address already in the queue, the new request is held until the first one completes, and then added to the end of the main queue. No request is rejected until there are 100 requests from a single IP address. So each IP address competes against itself, and no one IP can hog the system.

A typical overload comes when someone searches for an unusual topic and flips through many pages of search results quickly. This can result in the rating engine having fifty or so new sites to examine within a few seconds. Those requests will be processed one at a time until the rating engine catches up.

This queuing system held up well when someone tried a test where they fed a huge list of sites into the API without waiting for completion of any of their requests. The rating engine ran busily for a week, but they were only tying up one rating process, and it didn't affect other users at all.

There are no adjustment parameters. It just runs. It's not perfect, but it deals well with legitimate transients and with abuse from a small number of IP addresses.

This application is temporarily over its serving quota. Please try again later.

I'm actually not super sure is this is a single-serving joke site like isthemissiononfire.com, or if this is just a hilarious coincidence.

This is fast becoming my go-to tool for any sort of rate limiting at the application level.