Good writeup (with great diagrams). And a reminder how much we get out of the box with Elixir/Erlang.
Queues don't fix overload (2014)
21–30 of 156 posts
Re: Queues don't fix overload (2014)
#22This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…
The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorithm just adds servers magically when your load gets high.
Of course the system still has limits. But if you go from zonal to regional to global autoscaling, and you architect specifically to scale up each point in the system using autoscaling, it kind of doesn't have a limit? (that any one product would hit, at a global level)
In the past I have spun up a duplicate distributed system in another region and just split the traffic between the two. IaC is crazy.
Re: Queues don't fix overload (2014)
#23Real life queues can be scary too. I think of how complicated Disney's fast pass system got https://www.youtube.com/watch?v=9yjZpBq1XBE. Luckily with software it is way easier to get more servers than it is to build more theme park rides.
Re: Queues don't fix overload (2014)
#24Another thing. Queues often lack prioritization of messages; for instance, the importance of a new user signup may be overlooked compared to an address update.
There is a structure for just this purpose which, funnily enough, is known as a "priority queue".
Re: Queues don't fix overload (2014)
#25This is fitting. I just spent 99 minutes trying to register my kid for summer camps. Eventually got a “payment has been processed redirecting to receipt” message… which errored out.
Re: Queues don't fix overload (2014)
#26So far as I know there is no theoretical alternative to load shedding or increasing handling capacity if your average request arrival rate is greater than your average request handling rate. At least, not if you want to handle every accepted request using a finite queue[1]. It would appear that with an unbounded queue every request will eventually be handled, but with an unbounded latency guarantee. Which appears equ…
> It would appear that with an unbounded queue every request will eventually be handled
Of course there is no such a thing as an unbounded queue.
Re: Queues don't fix overload (2014)
#27So far as I know there is no theoretical alternative to load shedding or increasing handling capacity if your average request arrival rate is greater than your average request handling rate. At least, not if you want to handle every accepted request using a finite queue[1]. It would appear that with an unbounded queue every request will eventually be handled, but with an unbounded latency guarantee. Which appears equ…
> It would appear that with an unbounded queue every request will eventually be handled, but with an unbounded latency guarantee.
This might or might not be true. Depending on your strategy for retrieving tasks from the queue, it is not necessarily the case that every request will eventually be handled. FIFO will give you that guarantee; a priority system won't.
Re: Queues don't fix overload (2014)
#28[deleted]
Re: Queues don't fix overload (2014)
#29Been doing a lot of work on electronics and I think Queue's are very similar in a system to capacitors. They smooth out load either by buffering new work/load or holding onto pending work/load...
Re: Queues don't fix overload (2014)
#30Another thing. Queues often lack prioritization of messages; for instance, the importance of a new user signup may be overlooked compared to an address update.
Funny, I always think handling my existing customers and making sure things work for them is more important than taking in new ones.