Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using load shedding to avoid overload

aws.amazon.com

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Using load shedding to avoid overload

#41

Post by noisy_boy »

tyingq wrote: post #6

The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…

This makes me think about APIs that face a variety of queries (probably not e-commerce where patterns are more predictable but think internal warehouse) - some queries may be more "expensive" than others (e.g. they query data that touches a big percentage of the underlying data based on criteria where database index is not feasible). What if instead of allowing all requests to go to the API directly, we let them pass through a grader, "good" queries (say ones that are using indexed columns) get a higher "grade" and then we put the requests in a priority queue sorted by grades and descending time (so if grades are same, newer queries get processed first). That becomes the work queue. Interested to hear what people think about it / if someone is using something similar and their experience with it.

Re: Using load shedding to avoid overload

#42

Post by lokar »

tome wrote: post #29
dyanacek wrote: post #26

Earlier quoted context omitted.

Yes! LIFO is a fantastic improvement. This suggestion is buried in the article a bit. Maybe I should have elevated it, or maybe broken it into more pieces. There’s so much to talk about on this topic. But yeah, LIFO is totally “This one weird trick that will make your service bulletproof to overload! Chaos Monkeys hate it!”

I'm really confused about how and why this works. Why is it a good idea to keep really old requests unhandled? I feel like I must be missing something obvious.

Consider the two steady states:

You are keeping up and the queue is mostly empty, the order does not matter

You are not keeping up, the queue is growing. If nothing changes, the age of items removed from the front will grow and grow, and eventually all be timeouts or abandoned.

Re: Using load shedding to avoid overload

#43

Post by ozzythecat »

shukantpal wrote: post #39
ozzythecat wrote: post #18

Earlier quoted context omitted.

> You can always buy your way out of overload. Not really. My service may depend on other services that I have no control over. Perhaps I have extra money to scale up my own service, but those other services may be owned by different teams or organizations entirely.

With money, you can in-house those services and scale them up if those "other" organizations won't scale up.

With infinite money, sure. Realistically you won’t have the budget. And even if it’s “in house”, it’s going to be some service run by another team or organization within your company that has their own priorities, roadmap, and deliverables. They’re not just going to scale up their service because you asked them. They basically don’t have to do anything for you at all. And even if they agree - everything has a cost. There’s simply the overhead of even sending an email or a Slack message and setting up a meeting, getting people ramped up on your use case, why you need them to scale, for how long, for what use cases, etc etc. everything has a cost and money and budget are always constrained.

Re: Using load shedding to avoid overload

#44

Post by tome »

lokar wrote: post #42
tome wrote: post #29

Earlier quoted context omitted.

I'm really confused about how and why this works. Why is it a good idea to keep really old requests unhandled? I feel like I must be missing something obvious.

Consider the two steady states: You are keeping up and the queue is mostly empty, the order does not matter You are not keeping up, the queue is growing. If nothing changes, the age of items removed from the front will grow and grow, and eventually all be timeouts or abandoned.

Oh, I think I see. With a queue the failure state is that everything times out. With a stack you risk sacrificing only the oldest requests and keep the newest alive.

Re: Using load shedding to avoid overload

#45

Post by lokar »

tome wrote: post #44
lokar wrote: post #42

Earlier quoted context omitted.

Consider the two steady states: You are keeping up and the queue is mostly empty, the order does not matter You are not keeping up, the queue is growing. If nothing changes, the age of items removed from the front will grow and grow, and eventually all be timeouts or abandoned.

Oh, I think I see. With a queue the failure state is that everything times out. With a stack you risk sacrificing only the oldest requests and keep the newest alive.

And you end up not servicing some random sample that get buried in the stack before you can get to them.

Re: Using load shedding to avoid overload

#46

Post by withinboredom »

noisy_boy wrote: post #41
tyingq wrote: post #6

The article does mention prioritization, but doesn't mention my favorite pattern with this. A priority queue that favors end users "farther down the process" is nice for load shedding. Like, for an ecommerce site, being able to prioritize users in the checkout path first, not-in-checkout but non-empty cart second, etc. Or prioritizing features in a similar way. Turning off, for example, "people that bought this, boug…

This makes me think about APIs that face a variety of queries (probably not e-commerce where patterns are more predictable but think internal warehouse) - some queries may be more "expensive" than others (e.g. they query data that touches a big percentage of the underlying data based on criteria where database index is not feasible). What if instead of allowing all requests to go to the API directly, we let them pass…

TBH, the specific approach sounds like a bad one. You can easily swamp a server with small requests and the big ones will never run, which are probably important. It would just incentivize engineers to write smaller queries while your older (Unmaintained ones) ones would never run.

Same with deprioritizing queries that are older in the queue. You’d end up deprioritizing people who’ve already been waiting, so they’d likely try again instead of waiting.

Thus you end up with the feedback loop this article talks about, where you’re amplifying the problem instead of addressing it.

Re: Using load shedding to avoid overload

#47

Post by shukantpal »

ozzythecat wrote: post #43
shukantpal wrote: post #39

Earlier quoted context omitted.

With money, you can in-house those services and scale them up if those "other" organizations won't scale up.

With infinite money, sure. Realistically you won’t have the budget. And even if it’s “in house”, it’s going to be some service run by another team or organization within your company that has their own priorities, roadmap, and deliverables. They’re not just going to scale up their service because you asked them. They basically don’t have to do anything for you at all. And even if they agree - everything has a cost. T…

That's why the "with money" not "with little money"
Post reply on HN