11–20 of 156 posts
https://en.wikipedia.org/wiki/Little's_law
Additionally, here is a great talk on queuing theory and load shedding. One argument this talk makes is that autoscaling is not the silver bullet you think it is (similar to queues).
The other thing to bear in mind about queues is that once they start showing of symptoms of something being wrong, collapse might be just around the corner or it might not be depending on the nature of the load. When congestion spikes start showing it is helpful to know some queuing theory to estimate how close the situation is to eating someone's weekend. Congestion collapses are an interesting time because most peo…
Depends. An overflowing queue that drops jobs from the front while backend chugs along as fast as it can, is for many cases a better outcome than the backend overloading and everything grinding to a halt. Compare it to a physical service desk. The one that has a queue, serves one person at a time, and people arriving will try again another day if the queue is too long. The one without queue has people fistfighting ov…
This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…
If you're building an API, you don't care how your clients cope with your backpressure. You just want to avoid overloading your services, and queues will indeed not help you with that past a certain point, whereas backpressure probably will. Or at least you can scale backpressure much more cheaply than you can scale queues.
This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…
Queues are great, but can lead to catastrophic failure if you don't have a good way of handling the queue, so making an active choice about how you handle overload is part of designing a robust and resilient system.
Trading off new requests for current requests is, in my experience, a valid strategy for eCommerce for example. We called it "quenching".
If the queue can withstand more simultaneous load than the processing service, they basically do.
> Queues Don't Fix Overload If the queue can withstand more simultaneous load than the processing service, they basically do.
Edit: I should add that standing queues serve no purpose. If the demand is such that you're persistently queuing for a long time without the queue ever going idle, your transition between queuing and load-shedding is in the wrong place, and you should increase load-shedding, because you're adding latency for no reason.
It's that the people making changes don't have a decent understanding of the system they are trying to fix. If you don't actually know what the problem is, your fix is not likely to work.
I have seen groups put huge amounts of work into a "fix" for a system when they are only really guessing at what the problem is. (Besides queues, people also seem to like adding "caches" -- often stateful and with no validation strategy so not really caches -- or really any kind of wrapper or shim.)
I think it is really useful to raise awareness of this kind of thing, but I think the article could put it a little better. For one thing, queues can "fix overload" depending on what you mean by that. They don't increase system capacity but can let a system handle bursty usage better.
Fred is a big Erlang advocate from 10+ years ago.
He's written multiple books on Erlang, from his real-world experience, of using it to build Heroku (Erlang underpins much of the original Heroku platform and large pieces still today).
Much if his writing is influenced by said experience and Erlang native queues.