21–30 of 69 posts
A non-general solution that might work for most of us is often goes by the name "shopper prioritization". Basically if you have a site that does lots of processing (search, orders, etc) then a pretty effective strategy is often to show a generic static "down for maintenance, try again" page. This requires vastly fewer resources to serve since the page can be sitting in web server ram and doesn't result in rpc calls t…
Earlier quoted context omitted.
But by this point the request has already been accepted and processed. It's also using a rolling average, which means you'll be "killing" connections that otherwise wouldn't need to be killed.
Well, there is no exact threshold what "needs to bee killed". The higher the load the longer it takes to process requests. At some point you rather want to drop additional requests instead of making things even slower for everybody. This is what happens here. Think of it like this: When the server is already darn slow at a load of 10 we don't want to process more requests because we know at values over 10 it gets unb…
In other words, let's find the theoretical limit to what your web server can handle in terms of concurrent requests. Let's throw them all at your application. In this scenario, the check-load-then-die code is worthless because the request will never get there. It doesn't solve the problem as shown in the OP. What it does is mitigate the effect of the application on the total load of the web server. Which is another problem altogether.
Also:
> drop additional requests instead of making things even slower for everybody. This is what happens here.
Not exactly. The request completes like any HTTP request would. That you're die()ing out doesn't change that, at least from the perspective of the web server.
As with the authors example, when requests start stacking up in a process, we also potentially increase the memory load on the GC. It's also possible to cause a sort of pathological case, where the object allocation lives long enough to get promoted to the older generations, and then die, requiring almost constant major GC's to occur to free the promoted objects.
I worked on one system, where all performance testing was done in a lab environment under perfect conditions, and external resources all taking less than 1ms to respond. However, when deployed to a production network, it consistently performed abysmally, losing an order of magnitude of performance to having more outstanding requests being processed, and the associated GC load.
Newer GC's do appear to handle object promotion much better, but it's one more thing to consider, where some GC tuning and reduction in garbage can cause an order of magnitude increase in performance.
Anyway's, one more thing to consider.
Earlier quoted context omitted.
You also have the issue that you might be dropping new requests, but there's so much backlog that old requests eventually time out in droves, too. You might end up spending a lot of CPU cycles doing partial work on some requests, only to eventually see them hit their deadline. That's one of the reasons why Facebook has a mechanism that switches their FIFO queues to LIFO, once the server has passed a certain load/late…
Thereby inadvertently rewarding people who mash their reload button when the app/site is slow :)
That is, errors in internetworked systems can happen at a level high enough that machines can't decide if it is sane to retry or not, they should simply bubble the error all the way back to the user and let her press F5 if she wants to.
Modern UIs might hide this, but users well know the web is flaky and understand it in their bones. So when they press reload they are sending important information.
https://www.npmjs.com/package/toobusy-js
It's worth checking out - and helps you achieve this goal without necessarily having to come up with a req/s magicnum; instead you measure actual event loop latency to detect a slowdown.
This extremely simple solution works nicely for us: $load=file_get_contents('/proc/loadavg')[0]; if ($load>10) die('Server load currently too high.'); So when the load gets too high, the server will not procss the request. This will bring the server load down. An easy self regulating system.
Most solutions to the thundering herd problem involve rate limiting at the load balancer, or a finer-grained heuristic to ensure that each server does as much work as it can, even under load.
Earlier quoted context omitted.
You also have the issue that you might be dropping new requests, but there's so much backlog that old requests eventually time out in droves, too. You might end up spending a lot of CPU cycles doing partial work on some requests, only to eventually see them hit their deadline. That's one of the reasons why Facebook has a mechanism that switches their FIFO queues to LIFO, once the server has passed a certain load/late…
Thereby inadvertently rewarding people who mash their reload button when the app/site is slow :)
A classical solution is to use a control algorithm (for example PI or PID) to avoid having to estimate the overload threshold: the server will just start shedding a fraction of the requests to maintain some metric (CPU utilization, latency, queue length, ...) below some target. See for example [1]. This is about background jobs, but the same principles apply to load shedding. [1] http://folk.ntnu.no/skoge/prost/proce…
I developed a library to guard against server overload. It automatically gauges the capacity and rejects requests exceeding it. http://iheartradio.github.io/kanaloa/docs/theories.html