Using load shedding to avoid overload
aws.amazon.com
Using load shedding to avoid overload
1–10 of 47 posts
Re: Using load shedding to avoid overload
#2It is also a concept on electricity grid management: https://en.wikipedia.org/wiki/Rolling_blackout
Re: Using load shedding to avoid overload
#3This is a master class. So many good tidbits.
Re: Using load shedding to avoid overload
#4I think the blog post about load shedding by Netflix has some good examples.
"Keeping Netflix Reliable Using Prioritized Load Shedding" https://netflixtechblog.com/keeping-netflix-reliable-using-p...
Re: Using load shedding to avoid overload
#5It is also a concept on electricity grid management: https://en.wikipedia.org/wiki/Rolling_blackout
Indeed. This is the first I've heard the term used outside electric grid management.
Re: Using load shedding to avoid overload
#6The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding.
Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc.
Or prioritizing features in a similar way. Turning off, for example, "people that bought this, bought that", under load.
Re: Using load shedding to avoid overload
#7Since we are using the power abstraction, why not instead of load shedding, start up secondary machines servicing, which are slower, but could handle a bunch of simultaneous requests for some clients, or even tertiary ones?
That could at least postpone the load shedding, or handle localised surges in service demand due to user behaviour more effectively.
Re: Using load shedding to avoid overload
#8Since we are using the power abstraction, why not instead of load shedding, start up secondary machines servicing, which are slower, but could handle a bunch of simultaneous requests for some clients, or even tertiary ones? That could at least postpone the load shedding, or handle localised surges in service demand due to user behaviour more effectively.
The article is in Germany for me so I've only read a part of it before I got tired of trying to read German, but I think they mention as one of the first things that what you describe, preventing an overload in the first place, is their primary strategy.
The request dropping (less jargon-y and more descriptive than "load shedding", which refers to consuming more rather than less) only kicks in when that fails or isn't fast enough.
Re: Using load shedding to avoid overload
#9> Goodput is the subset of the throughput that is handled without errors and with low enough latency for the client to make use of the response.
"Back in my day..." we used HPS (hits per second) and CPU load as the "goodput". A 'hit' was only recorded via an access log at the end of a successful requestresponse. Of course this doesn't measure whether the latency of the responses is unacceptable, just that there was a completed response... so that's where CPU Load came in. If you're doing a lot of HPS and load gets too high, you know latency is just going to get worse to the point of unavailability, so you start load shedding.
In our less-advanced old-school practice, load shedding was merely lowering the maximum requests per second option of the HTTP servers, waiting, and lowering again, until the load recovered. Our tools could reconfigure the server's settings without restarting, but required issuing an admin command to the server and "waiting in line" while the server processed all the other CPU requests under load, which meant the problem might continue for longer than we'd like.
To get around "waiting in line", you would use a transparent network filter (either a proxy, or IPTables) to load-shed. This can be called a "middlebox" solution. But sometimes termination of connections isn't feasible without "dirty" terminations that might cause bugs in clients or servers... and that's [one of the reasons] why people hate middleboxes. But they're great when they work!
You would also probably use Apdex today instead of CPU Load. If your API has contractual minimum latency guarantees, you'd skip Apdex and just use latency itself.
Re: Using load shedding to avoid overload
#10Since we are using the power abstraction, why not instead of load shedding, start up secondary machines servicing, which are slower, but could handle a bunch of simultaneous requests for some clients, or even tertiary ones? That could at least postpone the load shedding, or handle localised surges in service demand due to user behaviour more effectively.
Sometimes there's limits to how high you can scale without overloading dependencies. For example, the database that the service accesses might only support n connections and the service is already using approximately n connections.