Earlier quoted context omitted.
Your mistake is characterising all overloads as the same. Adding a queue to a system that is consistently overloaded won't solve the overload, but many overloads are temporary/bursty, like the Slashdot/HN effect. Queues absolutely do solve these kinds of overloads simply by increasing latency, assuming increased latency is an acceptable choice in your context, of course.
Maximum acceptable response times for websites is a few seconds. The HN/Slashdot effect lasts for minutes to hours. That time scale is way too long for a queue to be effective.
Queues don't fix overload (2014)
91–100 of 156 posts
Re: Queues don't fix overload (2014)
#92It's extremely easy to introduce a backpressure mechanisms into your tech stack by using Go as a valve. If you can arrange data flow through a Go process, even if it's just a small tool inside your PHP/Rust/Javascript dream stack, then you can get backpressure done in about 5 lines of code: func valve[T any](ch chan The other half of the valve is a goroutine that reads from `ch` and pushes values further down your pi…
Re: Queues don't fix overload (2014)
#93This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…
For anyone reading the comments looking for more helpful guidance, the most generally helpful advice you will get is to start by measuring. Your measurements should help you identify bottlenecks, but the answer of how to fix the issue is dependent on your problem's constraints. In fact, there's such a massive number of dimensions to the solution space of the general problem of overloaded systems/services that it is actually ridiculous for the author to try and write a single ~2k words article explaining how to solve them all.
If you're hosting some non-critical service (which is clearly the case for the author), then maybe load-shedding is appropriate. If you're making a critical control system on the other hand, load-shedding (allowing system failure by design) is absurd. If you are already set up well for scaling (can be a big up-front investment) and have more money than time, maybe horizontal scaling is a no-brainer.
All those examples ignore the elephant in the room, which is the single root cause performance problem you're likely to discover after measuring your system and analyzing the results (good analysis is equally important to measurement, especially in large and/or distributed systems). This can range from something that is very expensive and possibly impractical to fix (ie: entire backend written in python), to a simple tweak (ie: fixing a bad database query) that drastically improves system performance with no downsides. The degree to which a system and it's components were designed by engineers not carelessly throwing away performance at the altar of "premature optimization is the root of all evil" can also have a profound impact.
Re: Queues don't fix overload (2014)
#94The other thing to bear in mind about queues is that once they start showing of symptoms of something being wrong, collapse might be just around the corner or it might not be depending on the nature of the load. When congestion spikes start showing it is helpful to know some queuing theory to estimate how close the situation is to eating someone's weekend. Congestion collapses are an interesting time because most peo…
Hey, can you recommend something one might read to get up to speed on queuing theory? I certainly am not aware of it, but work with queues.
Re: Queues don't fix overload (2014)
#95Earlier quoted context omitted.
No, load shedding and back pressure present you trade-offs to deal with an overloaded system. Queues don't. Queues just present you problems. If you take an overloaded system and add a queue, every single feature either gets worse or doesn't get any better. And people like to deny this, and pretend that queues will help. They absolutely help with a lot of things but they do nothing but harm in front of an overloaded…
Your mistake is characterising all overloads as the same. Adding a queue to a system that is consistently overloaded won't solve the overload, but many overloads are temporary/bursty, like the Slashdot/HN effect. Queues absolutely do solve these kinds of overloads simply by increasing latency, assuming increased latency is an acceptable choice in your context, of course.
What isn't happening on your example is a queue improving anything on a overloaded system.
Queues are immensely helpful for all kinds of problems. Just not for overload.
Re: Queues don't fix overload (2014)
#96Earlier quoted context omitted.
I learned about this stuff in grad school. The course wasn't mandatory for everyone but my supervisor made it mandatory for me due to the nature of the research I was doing: "Computer Systems and Performance Evaluation". It was basically focused on queuing theory and state space modelling. Reading through this whole discussion thread really makes me want to dig up my old notes and whip up a blog post with a Jupyter n…
If I were you I'd consider using https://simpy.readthedocs.io/en/latest/ inside a Jupyter notebook.
Re: Queues don't fix overload (2014)
#97Earlier quoted context omitted.
The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…
If I let my customers railroad me into running more servers to fulfill their “needs” then I may transition into losing money on my business. That’s not a solution. Needs is in scare quotes because a lot of traffic comes from misunderstanding or laziness from customers or from other divisions. Try as we might, nearly all of the improvements in capacity per customer on my project in the last six months have come from t…
Re: Queues don't fix overload (2014)
#98Earlier quoted context omitted.
Your mistake is characterising all overloads as the same. Adding a queue to a system that is consistently overloaded won't solve the overload, but many overloads are temporary/bursty, like the Slashdot/HN effect. Queues absolutely do solve these kinds of overloads simply by increasing latency, assuming increased latency is an acceptable choice in your context, of course.
After the overload fixes itself somehow, a queue absolutely solves the availability problem. Yep. If the overload doesn't last for too long. What isn't happening on your example is a queue improving anything on a overloaded system. Queues are immensely helpful for all kinds of problems. Just not for overload.
Re: Queues don't fix overload (2014)
#99Earlier quoted context omitted.
If I were you I'd consider using https://simpy.readthedocs.io/en/latest/ inside a Jupyter notebook.
Hey that's awesome! Unfortunate that I'm going to get even more confused now about whether I'm talking about SimPy or SymPy but that's life :)
Re: Queues don't fix overload (2014)
#100Earlier quoted context omitted.
The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…
Autoscaling is not going to help if you are IO-bound in your database. One point of the article is you have to identify your bottleneck before you can make sensible design choices.