Using load shedding to avoid overload
21–30 of 47 posts
Re: Using load shedding to avoid overload
#22The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…
This is a great way to describe it! I gave a similar example of pagination and how the later pages might be better to prioritize over initial pagination requests, but your example is a nicer illustration. Thanks for that! There’s also someone I was talking to after writing the article who said they can fall back to statically rendered versions of certain pages on Amazon.com during overload. The trick is to have a pag…
Ah, yes, you're right...I missed the pagination example fitting that pattern.
"If a feature on the site fails to render successfully or on time, it’s left off of the page. Critical functionality can be left off, so it’s a judgement call on what’s allowed to fail the page render."
Oh, that's useful also, but I meant a step farther where the page doesn't ask for those widgets if (load > X). Which avoids calling it at all.
Re: Using load shedding to avoid overload
#23The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…
The generic goal I think you are describing is to maximize goodput.
Another simple approach is to serve queues (of like requests) in LIFO order.
Re: Using load shedding to avoid overload
#24I think the blog post about load shedding by Netflix has some good examples. "Keeping Netflix Reliable Using Prioritized Load Shedding" https://netflixtechblog.com/keeping-netflix-reliable-using-p...
I also like "Performance Under Load" by Eran Landau, William Thurston and Tim Bozarth (https://netflixtechblog.medium.com/performance-under-load-3e...) very much.
Re: Using load shedding to avoid overload
#25Earlier quoted context omitted.
This is a great way to describe it! I gave a similar example of pagination and how the later pages might be better to prioritize over initial pagination requests, but your example is a nicer illustration. Thanks for that! There’s also someone I was talking to after writing the article who said they can fall back to statically rendered versions of certain pages on Amazon.com during overload. The trick is to have a pag…
Ah, yes, you're right...I missed the pagination example fitting that pattern. "If a feature on the site fails to render successfully or on time, it’s left off of the page. Critical functionality can be left off, so it’s a judgement call on what’s allowed to fail the page render." Oh, that's useful also, but I meant a step farther where the page doesn't ask for those widgets if (load > X). Which avoids calling it at a…
Good point around avoiding the call in the first place. This is a very tricky topic, I’ve found. Things that try to guess the nuanced health of a dependency can lead to outages when they guess wrong. These circuit breakers are helpful if they’re right, but harmful if they’re wrong.
For example, say a service is backed by a partitioned cache cluster, where the data is hashed to a particular cache node. Now let’s say one node has a problem, causing requests to data that lives on that node to fail, but others to succeed. If a client is making requests for data that happens to live on all nodes (the client doesn’t know about these nodes by the way, it’s just an implementation detail of the service) and sees an increased error rate, should it start failing some requests? It could take a single partition outage and increase the scope of impact into a full outage.
Anyway I’ve been meaning to write an Amazon Builders’ Library article on this topic, or to convince someone else to do it (looking at you, Marc Brooker!)
Re: Using load shedding to avoid overload
#26The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…
The generic goal I think you are describing is to maximize goodput. Another simple approach is to serve queues (of like requests) in LIFO order.
Yes! LIFO is a fantastic improvement. This suggestion is buried in the article a bit. Maybe I should have elevated it, or maybe broken it into more pieces. There’s so much to talk about on this topic. But yeah, LIFO is totally “This one weird trick that will make your service bulletproof to overload! Chaos Monkeys hate it!”
Re: Using load shedding to avoid overload
#27The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…
Graceful degradation by turning off optional features and making the critical path (order completion funnel) as lightweight as possible by reducing costs (# of results fetched/scored, turning off reco etc) was the best way to scale systems seeing 3x higher peaks year over year; and 10x higher peak during flash sales compared to normal periods, especially without breaking the bank trying to provision that much capacity.
p.s. - btw, no cloud has that much elastic capacity.
Re: Using load shedding to avoid overload
#28Most real world services have dependencies, and load tests don't accurately represent the fact that the dependencies have other users and changes in performance of a dependency will dramatically change the performance of the service being tested.
That pretty much means you can't have hard coded '100 requests per second per instance'.
Instead, I suspect the future of load shedding is automatic maximum "goodput" tracking. For example, the load balancer can alternate between 90 and 100 parallel requests, and if more requests are completed with fewer parallel requests then start alternating between 80 and 90 to figure which of those is better...
Re: Using load shedding to avoid overload
#29Earlier quoted context omitted.
The generic goal I think you are describing is to maximize goodput. Another simple approach is to serve queues (of like requests) in LIFO order.
Yes! LIFO is a fantastic improvement. This suggestion is buried in the article a bit. Maybe I should have elevated it, or maybe broken it into more pieces. There’s so much to talk about on this topic. But yeah, LIFO is totally “This one weird trick that will make your service bulletproof to overload! Chaos Monkeys hate it!”
I'm really confused about how and why this works. Why is it a good idea to keep really old requests unhandled? I feel like I must be missing something obvious.
Re: Using load shedding to avoid overload
#30The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…
Graceful degradation by turning off optional features and making the critical path (order completion funnel) as lightweight as possible by reducing costs (# of results fetched/scored, turning off reco etc) was the best way to scale systems seeing 3x higher peaks year over year; and 10x higher peak during flash sales compared to normal periods, especially without breaking the bank trying to provision that much capacit…
I saw this comment and thought this exactly described what my $previous_company did. Then, I saw the username :-)