Queues don't fix overload (2014) • Hacker News Board

Re: Queues don't fix overload (2014)

#51

Post by naasking » Thu, Jan 18, 2024, 3:14 PM UTC

marcosdumay wrote: post #35

kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

No, load shedding and back pressure present you trade-offs to deal with an overloaded system. Queues don't. Queues just present you problems. If you take an overloaded system and add a queue, every single feature either gets worse or doesn't get any better. And people like to deny this, and pretend that queues will help. They absolutely help with a lot of things but they do nothing but harm in front of an overloaded…

Your mistake is characterising all overloads as the same. Adding a queue to a system that is consistently overloaded won't solve the overload, but many overloads are temporary/bursty, like the Slashdot/HN effect. Queues absolutely do solve these kinds of overloads simply by increasing latency, assuming increased latency is an acceptable choice in your context, of course.

Re: Queues don't fix overload (2014)

#52

Post by HPsquared » Thu, Jan 18, 2024, 3:16 PM UTC

mvf4z7 wrote: post #11

Basically Little's law. It is queues all the way down. https://en.wikipedia.org/wiki/Little's_law Additionally, here is a great talk on queuing theory and load shedding. One argument this talk makes is that autoscaling is not the silver bullet you think it is (similar to queues). https://www.youtube.com/watch?v=-oQl1xv0hDk

Little's law reminds me of the Continuous Stirred-Tank Reactor model in chemical engineering. ChemEng has a lot of methods of modelling complex systems, that can be carried over to other domains.

https://en.wikipedia.org/wiki/Continuous_stirred-tank_reacto...

Re: Queues don't fix overload (2014)

#53

Post by multimoon » Thu, Jan 18, 2024, 3:16 PM UTC

marcosdumay wrote: post #35

kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

No, load shedding and back pressure present you trade-offs to deal with an overloaded system. Queues don't. Queues just present you problems. If you take an overloaded system and add a queue, every single feature either gets worse or doesn't get any better. And people like to deny this, and pretend that queues will help. They absolutely help with a lot of things but they do nothing but harm in front of an overloaded…

As another user pointed out, this is entirely based on the duration of the overload. If the overload is short, then queues absolutely do help smooth this over for the user.

If demand is consistently exceeding capacity, the system will fail regardless which method is used to compensate (queues, shedding, back pressure, etc). Saying that a queues as a whole serve no purpose in an overload is only considering the catastrophic scenario, there is a lot more types of overloads that can happen depending on the system.

Re: Queues don't fix overload (2014)

#54

Post by sethammons » Thu, Jan 18, 2024, 3:16 PM UTC

jmull wrote: post #19

Queues aren't really the problem here. It's that the people making changes don't have a decent understanding of the system they are trying to fix. If you don't actually know what the problem is, your fix is not likely to work. I have seen groups put huge amounts of work into a "fix" for a system when they are only really guessing at what the problem is. (Besides queues, people also seem to like adding "caches" -- oft…

Ran into this at work last week. A team wanted to add an in-mem cache to speed things up. However, they have no way nor plan to measure its effectiveness. I asked how they will know the cache hit ratio or otherwise know how the queue is working and they pushed back that such monitoring would be work they weren't planning. Bananas.

Re: Queues don't fix overload (2014)

#55

Post by kqr » Thu, Jan 18, 2024, 3:17 PM UTC

dijit wrote: post #15

kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

I think it's the threshold of all people to truly understand that there's no solution to certain problems, only tradeoffs. Queues are great, but can lead to catastrophic failure if you don't have a good way of handling the queue, so making an active choice about how you handle overload is part of designing a robust and resilient system. Trading off new requests for current requests is, in my experience, a valid strat…

> Trading off new requests for current requests is, in my experience, a valid strategy for eCommerce for example. We called it "quenching".

I'm not sure in which direction the trade happens but it sounds like you're dropping older requests in favour of newer. I agree, this has worked well for me also. Surprisingly often the oldest item in the queue is the one for which service will be least valuable.

Re: Queues don't fix overload (2014)

#56

Post by bob1029 » Thu, Jan 18, 2024, 3:19 PM UTC

Ring buffers and clever batching abstractions can help get you much closer to ideal.

> When consumers are waiting on an advancing cursor sequence in the ring buffer an interesting opportunity arises that is not possible with queues. If the consumer finds the ring buffer cursor has advanced a number of steps since it last checked it can process up to that sequence without getting involved in the concurrency mechanisms. This results in the lagging consumer quickly regaining pace with the producers when the producers burst ahead thus balancing the system. This type of batching increases throughput while reducing and smoothing latency at the same time. Based on our observations, this effect results in a close to constant time for latency regardless of load, up until the memory sub-system is saturated, and then the profile is linear following Little’s Law [6]. This is very different to the “J” curve effect on latency we have observed with queues as load increases.

https://lmax-exchange.github.io/disruptor/disruptor.html#_ba...

Re: Queues don't fix overload (2014)

#57

Post by kqr » Thu, Jan 18, 2024, 3:19 PM UTC

For anyone interested in this subject, the book Performance Modeling and Design of Computer Systems: Queueing Theory in Action is really, really good. TFA introduced me to queueing theory and that book made me understand the subject better than anything else I have read.

Re: Queues don't fix overload (2014)

#58

Post by jesuslop » Thu, Jan 18, 2024, 3:22 PM UTC

They fix jitter.

Re: Queues don't fix overload (2014)

#59

Post by tonyarkles » Thu, Jan 18, 2024, 3:33 PM UTC

secondcoming wrote: post #47

0xbadcafebee wrote: post #22

Earlier quoted context omitted.

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…

That's fine until the issue lies with something that your auto-scaled instances talk to, e.g. Redis, Scylla, SQL DB. There are situations where auto-scaling to infinity makes things far worse.

100% that was my first thought when reading the GP comment. Autoscaling isn't a magic fix. Just like the article says, you need to find the red arrow first to figure out where the bottleneck is and whether or not that bottleneck is actually something within the auto-scaling context or not. You've pointed out a couple of good examples of potential bottlenecks. Another possibility is a downstream 3rd-party service. If you start auto-scaling because a downstream service is struggling, you're just going to end up hitting it even harder and essentially contributing to a DoS attack against them.

No silver bullets and all that. Still need to do engineering analysis to figure out what's actually happening before you start shooting.

Re: Queues don't fix overload (2014)

#60

Post by smaudet » Thu, Jan 18, 2024, 3:34 PM UTC

dijit wrote: post #15

kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

I think it's the threshold of all people to truly understand that there's no solution to certain problems, only tradeoffs. Queues are great, but can lead to catastrophic failure if you don't have a good way of handling the queue, so making an active choice about how you handle overload is part of designing a robust and resilient system. Trading off new requests for current requests is, in my experience, a valid strat…

I think part of the article is just the author venting that people make design decisions without understanding the ramifications.

Operating a queue as a buffer would be absolutely fine if there were also service level agreements implicit in every queue in operation - i.e. one queue reaches half capacity or is experiencing explosive growth, leading to reactive techniques such as spinning up larger queues/servers, alerting stakeholders to the performance issue(s), and possibly even dynamically rate-limiting end-users.

But this is a whole-system centric view of the design. What they're specifically bemoaning is the asinine focus on component-centric design - your widget/service performs slowly? Throw a queue at it (and wonder why you broke everything 100 features and 10x customer base later)!

For legitimately small problems, this is OK. But then, you accept that you aren't scaling, and you bake it into your design assumptions. And you sell that, explicitly, to the customer "hey this will only work for 1 thousand/1 million customers, if you want more we can talk".