Denying requests to avoid fulfilling requests
Using load shedding to avoid overload
11–20 of 47 posts
Re: Using load shedding to avoid overload
#12Related concept in a recent HN thread about a paper describing how to use randomization to increase system resiliance: https://news.ycombinator.com/item?id=28702821
Re: Using load shedding to avoid overload
#13Since we are using the power abstraction, why not instead of load shedding, start up secondary machines servicing, which are slower, but could handle a bunch of simultaneous requests for some clients, or even tertiary ones? That could at least postpone the load shedding, or handle localised surges in service demand due to user behaviour more effectively.
The article is in Germany for me so I've only read a part of it before I got tired of trying to read German, but I think they mention as one of the first things that what you describe, preventing an overload in the first place, is their primary strategy. The request dropping (less jargon-y and more descriptive than "load shedding", which refers to consuming more rather than less) only kicks in when that fails or isn'…
This official PDF should be in English: https://d1.awsstatic.com/builderslibrary/pdfs/using-load-she...
Re: Using load shedding to avoid overload
#14Since we are using the power abstraction, why not instead of load shedding, start up secondary machines servicing, which are slower, but could handle a bunch of simultaneous requests for some clients, or even tertiary ones? That could at least postpone the load shedding, or handle localised surges in service demand due to user behaviour more effectively.
The article is in Germany for me so I've only read a part of it before I got tired of trying to read German, but I think they mention as one of the first things that what you describe, preventing an overload in the first place, is their primary strategy. The request dropping (less jargon-y and more descriptive than "load shedding", which refers to consuming more rather than less) only kicks in when that fails or isn'…
You can change language in the footer, waaay down.
Re: Using load shedding to avoid overload
#15This is a great article but leaves out a key idea. Load shedding is really a key topic in cost utilization. You need load shedding to be able to serve closer to your capacity red line. You can always buy your way out of overload. Load shedding is a feature that, if you have it, allows you to more comfortably dial back your resources and serve closer to the limit.
Re: Using load shedding to avoid overload
#16The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…
This is a great way to describe it! I gave a similar example of pagination and how the later pages might be better to prioritize over initial pagination requests, but your example is a nicer illustration. Thanks for that!
There’s also someone I was talking to after writing the article who said they can fall back to statically rendered versions of certain pages on Amazon.com during overload. The trick is to have a page that is still useful!
And for the “turning off features” idea - this happens today on Amazon.com. If a feature on the site fails to render successfully or on time, it’s left off of the page. Critical functionality can be left off, so it’s a judgement call on what’s allowed to fail the page render.
Re: Using load shedding to avoid overload
#17Denying requests to avoid fulfilling requests
It’s a tricky topic because load shedding is a last resort that kicks in when there’s already a problem. So until auto scaling catches up or the issue is mitigated some other way, we try to make as many customers happy as possible, rather than making everyone equally unhappy.
Re: Using load shedding to avoid overload
#18This is a great article but leaves out a key idea. Load shedding is really a key topic in cost utilization. You need load shedding to be able to serve closer to your capacity red line. You can always buy your way out of overload. Load shedding is a feature that, if you have it, allows you to more comfortably dial back your resources and serve closer to the limit.
> You can always buy your way out of overload.
Not really. My service may depend on other services that I have no control over. Perhaps I have extra money to scale up my own service, but those other services may be owned by different teams or organizations entirely.
Re: Using load shedding to avoid overload
#19This is a great article but leaves out a key idea. Load shedding is really a key topic in cost utilization. You need load shedding to be able to serve closer to your capacity red line. You can always buy your way out of overload. Load shedding is a feature that, if you have it, allows you to more comfortably dial back your resources and serve closer to the limit.
Agreed - utilization is an important consideration here. The capacity red line will still be there, but when load shedding is effective, the impact of crossing that red line is les. It’d be an error rate linearly proportionally to the excess, rather than the service falling off a cliff. But for the services this is talking about, neither case is okay, so we put a ton of emphasis on auto scaling models to make sure we don’t get into the situation.
A key sort of “continuation” to this article is the one on fairness: https://aws.amazon.com/builders-library/fairness-in-multi-te... . This gets into the topic of utilization a bit more.
But you’re right - good load shedding gives a business a tool to make an easier trade off when it comes to capacity management. A slight error rate until autoscaling kicks in is an easier pill to swallow than a worse outage.
Re: Using load shedding to avoid overload
#20> Goodput is the subset of the throughput that is handled without errors and with low enough latency for the client to make use of the response. "Back in my day..." we used HPS (hits per second) and CPU load as the "goodput". A 'hit' was only recorded via an access log at the end of a successful request response. Of course this doesn't measure whether the latency of the responses is unacceptable, just that there was…
These remain as great techniques! Even iptables like you mention - it’s extremely good at cheaply shedding new handshakes, vs later on in processing the request. You lose a little visibly, but it’s a powerful outer “layer of the onion”.
And good callout on middle boxes. Even high level abstraction ones like Amazon API Gateway. In fact this is my favorite feature of it. API Gateway can reject a very high rate of excess traffic for a small overloaded service behind it.