Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preventing server overload: limit requests being processed

evanjones.ca

Post reply on HN
Print view View original

1–10 of 69 posts

Re: Preventing server overload: limit requests being processed

#3

Post by abritinthebay »

This is basically how Hapi works in the NodeJS world.

If the event loop is too long (you configure this) it means you're doing too much! So it issues 503s to all new traffic until the server can cope with the requests it's receiving and get the loop time time.

Works very well and you can configure it to do the same with memory limits as well.

Doesn't fix all issues, but it's a good way to keep your site usable by some users at least.

Re: Preventing server overload: limit requests being processed

#4

Post by makmanalp »

The thing that trips me up along with request caps a lot is timeouts (briefly mentioned). There are so many network / API / retry timeouts set to arbitrarily large amounts (e.g. 30 seconds) that it's hard to predict how they'll interact, and often you don't know that some of these exist and at which layers they exist (application layer, network layer, DNS, etc).

How you often see this is that your request limiter keeps booting new requests (FIFO/drop tail), and then when you check your currently processing requests, they seem to be all in some busy loop, and hopefully you have some means of checking why. You check, and you find that they're all querying a service that accepts the request but doesn't respond ever (e.g. server down behind a proxy), but for some reason the timeout on the client side is set to 30 seconds so stuff keeps hanging until it's killed.

Anyway, this is to say that in any sufficiently complex networked system scenario, it helps to review all these timeouts holistically and in the context of your application.

Re: Preventing server overload: limit requests being processed

#5

Post by kyrra »

[I'm a Google employee, opinions are my own]

The GCP team did a blog post similar to this about using Load Shedding to survive a spike[0]. It's definitely a great way to survive a sudden spike in traffic, but not optimal, as serving errors is also bad (unless your clients have some sort of retry mechanism in-place). But if your clients do have a retry mechanism in place (especially one that has a backoff time on retries), it can be a great way to handle high loads.

[0] https://cloudplatform.googleblog.com/2016/12/using-load-shed...

Re: Preventing server overload: limit requests being processed

#6

Post by tyingq »

Haproxy is very flexible for this sort of thing. You can, for example, limit concurrent connections to backends, and place requests in a queue with a specific timeout. Not at all automatic, but at least all the right knobs are there.

Edit: Related, Fedex's tracking is down on their website right now. It's returning quickly, though, with this in the JSON: "Service com.fedex.nxgen.trck.v8.ientities.TrckInterface is busy, max invoke limit reached"

Re: Preventing server overload: limit requests being processed

#8

Post by spyspy »

kyrra wrote: post #5

[I'm a Google employee, opinions are my own] The GCP team did a blog post similar to this about using Load Shedding to survive a spike[0]. It's definitely a great way to survive a sudden spike in traffic, but not optimal, as serving errors is also bad (unless your clients have some sort of retry mechanism in-place). But if your clients do have a retry mechanism in place (especially one that has a backoff time on retr…

My team moved out entire platform to AppEngine to deal with the traffic spikes we get (among other reasons). It's worked out quite well I must say. It's also cool that if you know you have a spike coming that even AE can't handle, you can automatically tell the service to scale itself a few minutes beforehand in anticipation and then scale itself down later after it's passed.

Re: Preventing server overload: limit requests being processed

#10

Post by necrodome »

abritinthebay wrote: post #3

This is basically how Hapi works in the NodeJS world. If the event loop is too long (you configure this) it means you're doing too much! So it issues 503s to all new traffic until the server can cope with the requests it's receiving and get the loop time time. Works very well and you can configure it to do the same with memory limits as well. Doesn't fix all issues, but it's a good way to keep your site usable by som…

Not for some users though, usable for some requests. Users will still be annoyed when they can't complete actions, because some of their requests are failing randomly.
Post reply on HN