Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preventing server overload: limit requests being processed

evanjones.ca

Post reply on HN
Print view View original

11–20 of 69 posts

Re: Preventing server overload: limit requests being processed

#11

Post by TekMol »

This extremely simple solution works nicely for us:

    $load=file_get_contents('/proc/loadavg')[0];
    if ($load>10) die('Server load currently too high.');
So when the load gets too high, the server will not procss the request. This will bring the server load down. An easy self regulating system.

Re: Preventing server overload: limit requests being processed

#12

Post by anonacct37 »

A non-general solution that might work for most of us is often goes by the name "shopper prioritization".

Basically if you have a site that does lots of processing (search, orders, etc) then a pretty effective strategy is often to show a generic static "down for maintenance, try again" page. This requires vastly fewer resources to serve since the page can be sitting in web server ram and doesn't result in rpc calls throughout the whole stack.

You can also do something like prioritize logged in users over casual browsers.

Since most http clients are smart clients you can even do some stuff with rngs and timeouts to selectively let people who are willing to wait around back into the site.

Of course if you're an ad-tech company it's likely that serving an error is about as fast as serving something successfully.

Re: Preventing server overload: limit requests being processed

#13

Post by nkozyra »

TekMol wrote: post #11

This extremely simple solution works nicely for us: $load=file_get_contents('/proc/loadavg')[0]; if ($load>10) die('Server load currently too high.'); So when the load gets too high, the server will not procss the request. This will bring the server load down. An easy self regulating system.

But by this point the request has already been accepted and processed. It's also using a rolling average, which means you'll be "killing" connections that otherwise wouldn't need to be killed.

Re: Preventing server overload: limit requests being processed

#14

Post by TekMol »

nkozyra wrote: post #13
TekMol wrote: post #11

This extremely simple solution works nicely for us: $load=file_get_contents('/proc/loadavg')[0]; if ($load>10) die('Server load currently too high.'); So when the load gets too high, the server will not procss the request. This will bring the server load down. An easy self regulating system.

But by this point the request has already been accepted and processed. It's also using a rolling average, which means you'll be "killing" connections that otherwise wouldn't need to be killed.

Well, there is no exact threshold what "needs to bee killed". The higher the load the longer it takes to process requests. At some point you rather want to drop additional requests instead of making things even slower for everybody. This is what happens here.

Think of it like this: When the server is already darn slow at a load of 10 we don't want to process more requests because we know at values over 10 it gets unbearable.

Re: Preventing server overload: limit requests being processed

#15

Post by ot »

A classical solution is to use a control algorithm (for example PI or PID) to avoid having to estimate the overload threshold: the server will just start shedding a fraction of the requests to maintain some metric (CPU utilization, latency, queue length, ...) below some target.

See for example [1]. This is about background jobs, but the same principles apply to load shedding.

[1] http://folk.ntnu.no/skoge/prost/proceedings/acc04/Papers/035...

Re: Preventing server overload: limit requests being processed

#17

Post by puzzle »

makmanalp wrote: post #4

The thing that trips me up along with request caps a lot is timeouts (briefly mentioned). There are so many network / API / retry timeouts set to arbitrarily large amounts (e.g. 30 seconds) that it's hard to predict how they'll interact, and often you don't know that some of these exist and at which layers they exist (application layer, network layer, DNS, etc). How you often see this is that your request limiter kee…

You also have the issue that you might be dropping new requests, but there's so much backlog that old requests eventually time out in droves, too. You might end up spending a lot of CPU cycles doing partial work on some requests, only to eventually see them hit their deadline.

That's one of the reasons why Facebook has a mechanism that switches their FIFO queues to LIFO, once the server has passed a certain load/latency threshold.

Re: Preventing server overload: limit requests being processed

#19

Post by rrdharan »

puzzle wrote: post #17
makmanalp wrote: post #4

The thing that trips me up along with request caps a lot is timeouts (briefly mentioned). There are so many network / API / retry timeouts set to arbitrarily large amounts (e.g. 30 seconds) that it's hard to predict how they'll interact, and often you don't know that some of these exist and at which layers they exist (application layer, network layer, DNS, etc). How you often see this is that your request limiter kee…

You also have the issue that you might be dropping new requests, but there's so much backlog that old requests eventually time out in droves, too. You might end up spending a lot of CPU cycles doing partial work on some requests, only to eventually see them hit their deadline. That's one of the reasons why Facebook has a mechanism that switches their FIFO queues to LIFO, once the server has passed a certain load/late…

Thereby inadvertently rewarding people who mash their reload button when the app/site is slow :)
Post reply on HN