Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using load shedding to avoid overload

aws.amazon.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Using load shedding to avoid overload

#31

Post by tyingq »

tome wrote: post #29
dyanacek wrote: post #26

Earlier quoted context omitted.

Yes! LIFO is a fantastic improvement. This suggestion is buried in the article a bit. Maybe I should have elevated it, or maybe broken it into more pieces. There’s so much to talk about on this topic. But yeah, LIFO is totally “This one weird trick that will make your service bulletproof to overload! Chaos Monkeys hate it!”

I'm really confused about how and why this works. Why is it a good idea to keep really old requests unhandled? I feel like I must be missing something obvious.

I suppose the older requests are less likely to have actual people doing a page reload/retry that stacks up yet more demand.

Re: Using load shedding to avoid overload

#32

Post by dmurray »

tyingq wrote: post #6

The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…

> doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding.

The (fantastic) article suggests precisely this:

> let's say a service has two APIs: start() and end(). In order to finish their work, clients need to be able to call both APIs. In this case, the service should prioritize end() requests over start() requests

Re: Using load shedding to avoid overload

#33

Post by oriolid »

tome wrote: post #29
dyanacek wrote: post #26

Earlier quoted context omitted.

Yes! LIFO is a fantastic improvement. This suggestion is buried in the article a bit. Maybe I should have elevated it, or maybe broken it into more pieces. There’s so much to talk about on this topic. But yeah, LIFO is totally “This one weird trick that will make your service bulletproof to overload! Chaos Monkeys hate it!”

I'm really confused about how and why this works. Why is it a good idea to keep really old requests unhandled? I feel like I must be missing something obvious.

It guarantees that in overload situation the requests that get handled are handled quickly. In the same situation a FIFO would grow until all requests are really slow or time out without increasing throughput. The reason to keep old requests in LIFO instead of dropping then right away is that they can be served when load drops, just in case there's still someone waiting for the page to load.

Re: Using load shedding to avoid overload

#34

Post by kawsper »

vinay_ys wrote: post #27
tyingq wrote: post #6

The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…

Graceful degradation by turning off optional features and making the critical path (order completion funnel) as lightweight as possible by reducing costs (# of results fetched/scored, turning off reco etc) was the best way to scale systems seeing 3x higher peaks year over year; and 10x higher peak during flash sales compared to normal periods, especially without breaking the bank trying to provision that much capacit…

> turning off reco

What is reco?

Re: Using load shedding to avoid overload

#35

Post by [deleted] »

tome wrote: post #29
dyanacek wrote: post #26

Earlier quoted context omitted.

Yes! LIFO is a fantastic improvement. This suggestion is buried in the article a bit. Maybe I should have elevated it, or maybe broken it into more pieces. There’s so much to talk about on this topic. But yeah, LIFO is totally “This one weird trick that will make your service bulletproof to overload! Chaos Monkeys hate it!”

I'm really confused about how and why this works. Why is it a good idea to keep really old requests unhandled? I feel like I must be missing something obvious.

[deleted]

Re: Using load shedding to avoid overload

#36

Post by iampims »

kawsper wrote: post #34
vinay_ys wrote: post #27

Earlier quoted context omitted.

Graceful degradation by turning off optional features and making the critical path (order completion funnel) as lightweight as possible by reducing costs (# of results fetched/scored, turning off reco etc) was the best way to scale systems seeing 3x higher peaks year over year; and 10x higher peak during flash sales compared to normal periods, especially without breaking the bank trying to provision that much capacit…

> turning off reco What is reco?

Recommendations.

Re: Using load shedding to avoid overload

#37

Post by 8note »

tome wrote: post #29
dyanacek wrote: post #26

Earlier quoted context omitted.

Yes! LIFO is a fantastic improvement. This suggestion is buried in the article a bit. Maybe I should have elevated it, or maybe broken it into more pieces. There’s so much to talk about on this topic. But yeah, LIFO is totally “This one weird trick that will make your service bulletproof to overload! Chaos Monkeys hate it!”

I'm really confused about how and why this works. Why is it a good idea to keep really old requests unhandled? I feel like I must be missing something obvious.

A queue distributes the latency increase to all requests whereas a stack only increases the latency for some requests when you're overloaded.

This means that if you catch up to the incoming new requests, a queue keeps every request running slower due to the time spent in the queue.

The steady state stack on the other hand, gives the same couple items worse and worse latency, while all the new items go back to normal.

Chances are the long latency requests will be retried, and there's a roughly fixed number of items to be retried, so you don't have to worry too much about the ones stuck in the stack. For a queue, the retries lengthen the queue, and that waiting time is added to every item, making them more likely to retry too, lengthening the queue further etc

Having the stack leak items at the bottom gets the same benefit - you don't have to fulfil them, but if you can get the items back out of the stack quickly, it's still worth working on and completing them before the client needs to send a retry. The more of them you get in, the more 9s you get to look more like your p50 than your p100

Re: Using load shedding to avoid overload

#38

Post by ignoramous »

tyingq wrote: post #6

The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…

> A priority queue that favors end users "farther down the process" is nice for load shedding.

Another pattern from the networking world, one which Facebook wrote about, is CoDel aka Controlled Delay (switches tasks to lower timeouts when the work-queue begins to build-up) combined with adaptive LIFO (processes last-in tasks first when under load, but FIFO otherwise): https://queue.acm.org/detail.cfm?id=2839461

Re: Using load shedding to avoid overload

#39

Post by shukantpal »

ozzythecat wrote: post #18
jeffbee wrote: post #15

This is a great article but leaves out a key idea. Load shedding is really a key topic in cost utilization. You need load shedding to be able to serve closer to your capacity red line. You can always buy your way out of overload. Load shedding is a feature that, if you have it, allows you to more comfortably dial back your resources and serve closer to the limit.

> You can always buy your way out of overload. Not really. My service may depend on other services that I have no control over. Perhaps I have extra money to scale up my own service, but those other services may be owned by different teams or organizations entirely.

With money, you can in-house those services and scale them up if those "other" organizations won't scale up.

Re: Using load shedding to avoid overload

#40

Post by throwawaythekey »

Load shedding as a technique seems to make sense but to me it never seems to be the correct tool to reach for. Of course you need some basic load shedding to handle ddos type scenarios, but anything beyond that seems like overkill.

At (work) our services achieve high availability (99.99%) simply by being composed of reliable parts. Most effort is focused here and it seems to have a good payoff ratio.

When we do try to add anything like graceful degradation it seems to strongly hurt code readability/ease of understanding because it requires additional code branches and either a fair bit of boilerplate or extra abstractions.

This could be a result of the scale we are at/the maturity of our tools but I'm interested in if anyone else has some hands on experience.

Post reply on HN