Preventing server overload: limit requests being processed • Hacker News Board

Re: Preventing server overload: limit requests being processed

#61

Post by mnw21cam » Tue, Aug 08, 2017, 12:20 PM UTC

The main problem with this article is that it assumes that the only resource consumption is CPU/memory on the server. With that assumption, it makes complete sense to limit the number of concurrent connections.

However, if you do so, then your server can be brought to its knees by a single client opening that many connections, and then just being really slow in accepting the data down the network. Then the limited connections on the server are all sitting there doing nothing, and no new connections are allowed.

Re: Preventing server overload: limit requests being processed

#62

Post by kijin » Tue, Aug 08, 2017, 12:36 PM UTC

makmanalp wrote: post #4

The thing that trips me up along with request caps a lot is timeouts (briefly mentioned). There are so many network / API / retry timeouts set to arbitrarily large amounts (e.g. 30 seconds) that it's hard to predict how they'll interact, and often you don't know that some of these exist and at which layers they exist (application layer, network layer, DNS, etc). How you often see this is that your request limiter kee…

Heh, timeouts are the bane of my life as well as my most amusing source of extra income.

Client: I'm getting 504 Gateway Timeout errors on my site. Can you help?

Me: After how many seconds?

Client: About 30.

Me: Reduce the timeout to 20 seconds in your PHP-FPM pool configuration.

Client: But then I'll get even more timeouts!

Me: No you won't. While you're at it, reduce it even further to 10 seconds.

Client: What? That doesn't make any sense!

Me: Let's try 5 seconds then.

... Silence ...

Client: Hey, no more timeouts! But the widget on the sidebar is broken.

Me: That widget was trying to access an API that is currently down. Due to the long timeout and poor caching, it also took down your whole website. I heard you paid $cheapCompetitor to make the widget. If you'd like me to make you a better version by Saturday, that would be $XXX.

Re: Preventing server overload: limit requests being processed

#63

Post by evanj » Tue, Aug 08, 2017, 12:57 PM UTC

wutbrodo wrote: post #37

> Over Quota This application is temporarily over its serving quota. Please try again later. I'm actually not super sure is this is a single-serving joke site like isthemissiononfire.com, or if this is just a hilarious coincidence.

I use App Engine's static file serving to serve my site. It has a 1 GB/day bandwidth limit. It turns out Hacker News still moves a lot of traffic, so this is the first time I've crossed it! Oops. Billing is now enabled on this project.

Re: Preventing server overload: limit requests being processed

#64

Post by evanj » Tue, Aug 08, 2017, 1:09 PM UTC

mnw21cam wrote: post #61

The main problem with this article is that it assumes that the only resource consumption is CPU/memory on the server. With that assumption, it makes complete sense to limit the number of concurrent connections. However, if you do so, then your server can be brought to its knees by a single client opening that many connections, and then just being really slow in accepting the data down the network. Then the limited co…

This is a good point, although I don't think I said "connections", I said "requests". The definition of requests is going to vary significantly. Yes, very slow clients are yet another problem that extremely robust systems don't have to handle.

In my personal case, nearly all of my work has always been on "internal" services that are being used inside a single organization inside a data center, where this is rarely a problem. This is much more of an issue for services that are accessed over the public Internet, where you need to handle malicious attackers as well as users that have horrible connections.

In any case, I believe what I wrote still applies, but you do need to set appropriate timeouts (yet another parameter to tune; yuck!)

Re: Preventing server overload: limit requests being processed

#65

Post by evanj » Tue, Aug 08, 2017, 1:13 PM UTC

boulos wrote: post #57

For those asking, this seems to be hosted on App Engine and he's gone over his quota. App Engine's throttling is both a blessing and a curse (presumably he's using the free tier and never noticed this). You'll note that it's explicitly saying he's over quota not that it can't handle it ;). Disclosure: I work on Google Cloud.

Exactly right. I just enabled billing on this project. Oops!

Re: Preventing server overload: limit requests being processed

#66

Post by evanj » Tue, Aug 08, 2017, 1:15 PM UTC

peterwwillis wrote: post #33

In AWS, achieving peak theoretical performance requires queueing & pooling jobs of specific sizes. You are literally going to suffer costly bad performance if you don't implement and limit backlogs and concurrency. This is basically just "stacks and queues" in a general sense, most often seen as wait-and-retry in an application, and rate controls in a service. To do this and survive cascading failures you have to kno…

I agree that a universal solution sounds difficult. However, it seems like it should be possible to have something that works "well enough" for some (broad) class of servers. My example is TCP, which does a "good enough" job of controlling the flow of packets over a wide range of networks.

Re: Preventing server overload: limit requests being processed

#67

Post by evanj » Tue, Aug 08, 2017, 1:19 PM UTC

Animats wrote: post #35

I've used fair queuing to solve this problem. I have a site and API which rates other sites, checking them for ad links and trying to match them to real world business records. Inspecting a site takes 10 seconds to 2 minutes. The API, which is usually used from browser add-ons which tag web search results, lets anyone request the rating info for a site. Most requests are already cached and return immediately. Request…

Fair queuing is an interesting and complementary approach that I have not considered, thanks for the suggestion.

I usually work with "internal" systems that don't always have an obvious "user" identifier, but it would be fairly easy to add one of some sort (e.g. an application id or similar). However, for that to be useful, there still has to be some sort of limit. In your case, the limit seems to be 1 concurrent request per user, up to a maximum of 100. In the case of some internal system, like say a metrics aggregator, what is the correct limit? I'm not sure, and I think it would require tuning.

That said, given a reasonable limit, this would definitely ensure that the "misbehaving" application (e.g. the one flooding the metrics aggregator with packets) is the one that gets punished, rather than everyone else. I'll have to think about this more. Thanks!

Re: Preventing server overload: limit requests being processed

#68

Post by TekMol » Tue, Aug 08, 2017, 5:26 PM UTC

nkozyra wrote: post #22

TekMol wrote: post #14

Earlier quoted context omitted.

Well, there is no exact threshold what "needs to bee killed". The higher the load the longer it takes to process requests. At some point you rather want to drop additional requests instead of making things even slower for everybody. This is what happens here. Think of it like this: When the server is already darn slow at a load of 10 we don't want to process more requests because we know at values over 10 it gets unb…

My point is it's still processing the request. You're just shielding the web server from whatever the application holds. In other words, let's find the theoretical limit to what your web server can handle in terms of concurrent requests. Let's throw them all at your application. In this scenario, the check-load-then-die code is worthless because the request will never get there. It doesn't solve the problem as shown…

    find the theoretical limit to what your web server
    can handle in terms of concurrent requests. Let's
    throw them all at your application. In this scenario,
    the check-load-then-die code is worthless because
    the request will never get there.

If there is such a limit, then why not mitigate that by setting a max number of connections for the webserver?

    You're just shielding the web server from
    whatever the application holds.

True. But as I said: it works. Never seen our server get slow since we implemented this a few years ago. It kicks in when we get about 25x our normal traffic. Which happened less then once a year so far.

Re: Preventing server overload: limit requests being processed

#69

Post by jonhohle » Thu, Aug 10, 2017, 4:59 AM UTC

robbles wrote: post #56

jonhohle wrote: post #44

Earlier quoted context omitted.

Retries aren't the only option if you control (or can at least influence) the client. Fallback sources and sane (or customer preferential) defaults are some additional options when the primary source of truth is unavailable. Both of these bypass thundering heard situations where recovery can be difficult even if the application returns to full health. As a concrete example, when I worked at a large online retailer, w…

Why wouldn't you start with the cache? Isn't the consistent database only useful if you require it to be used, and don't fall back to a less consistent alternative?

Based on previous decisions made by customers that were processed within the lifetime of the same request, there was some probability that the cache could be out of date.

Typically, there was enough context within the request to make assumptions about the state of the authoritative store if data hadn't propagated to the cache in time. We eventually moved to using that initially.

edit: I should add, though, the order of the datasources wasn't critical to the strategy: have a primary data source, backup data source, use any data available in the request context, and finally err in the customer's favor. That's 4 chances to get things right, none of which include retrying.