Why disaster happens at the edges: An introduction to queue theory

Bitten by queues in every team I was at AWS:

> As a rule of thumb, target utilization below 75%

This is one good reason to rely on serverless. Simply outsource the problem to a system that knows how to handle this better.

> Steer slower workloads to paths with lower utilization

This is fraught with all sorts of perils [0], so take caution going down this valid but tricky route.

> Limit variance as much as possible when utilization is high

This is key, and one of the most elegant solutions to this problem I know of comes from a Facebook talk on CoDel + Adaptive LIFO. [1]

> Implement backpressure in systems where it is not built-in

Making downstream dependencies behave is never an option. Selectively isolating noisy neighbours, if possible, from other well behaved clients, tends to work well for multi-tenant systems [2], in addition to monitoring long work (by dropping work that takes forever) [3], or better yet, doing constant amount of work [4] (aka eliminating modes) [5].

> Use throttling and load shedding to reduce pressure on downstream queues.

One way is to impl admission control (ala Token Bucket) to minimise the impact of thundering herds. Though, clients do not take kindly to being throttled. [6]

A great article; many have been written at this point. Very many still have been burned by queues. Though, in my experience, without an exception, some component somewhere was always building up that backlog! [7][8][9]

[0] Interns with Toasters: How I taught people about Load Balancers, https://news.ycombinator.com/item?id=16894946

[1] Fail at scale: Controlling queue delays, https://blog.acolyer.org/2015/11/19/fail-at-scale-controllin...

[2] Worload isolation, https://aws.amazon.com/builders-library/workload-isolation-u...

[3] Avoiding insurmountable queue backlogs, https://aws.amazon.com/builders-library/avoiding-insurmounta...

[4] Constant work, https://aws.amazon.com/builders-library/reliability-and-cons...

[5] Cache, modes, and unstable systems, https://news.ycombinator.com/item?id=28344561

[6] Fairness in multi-tenant systems, https://aws.amazon.com/builders-library/fairness-in-multi-te...

[7] Using load shedding to avoid overload, https://aws.amazon.com/builders-library/using-load-shedding-...

[8] Treadmill: Precise load testing, https://research.fb.com/publications/treadmill-attributing-t...

[9] Cascading failures, https://sre.google/sre-book/addressing-cascading-failures/

Right, and now you're using the argument of the top level comment.

Look at the port. Is ingress consistently larger than egress? Then limit ingress.

Now look at the ocean. Is ingress consistently larger than egress? Then limit ingress.

Now look at the exporting ports. Is ingress consistently larger than egress? Limit ingress.

Now look at trucks going to the port. Is ingress consistently larger than egress? Limit ingress.

Now look at the factories that load trucks. Is ingress consistently larger than egress? Limit ingress.

----

Back pressure builds on looking at one queue and asking if arrivals come faster than departures go -- exactly what the top-level comment suggested.

Another random software engineer claiming a fundamental theoretical problem irrelevant...

Queuing theory deals with the stochastic behavior of a discreet event in at a single queue, and possibly multiple ones connected in some structured manner.

It's importance is that it can determine before hand what are the stochastic boundary of the system.

Your whole idea of 99 events and 101 events are so strawman that I don't know where to start poking at the holes...

For one thing, the rate of something has its definition, there is jitter or busyness. To say something has a rate is plainly a impractical concept. As there is nothing in the network that had steady rate...

Edit: I should stand corrected that the parent does not really miss the meaning of queuing theory. It was just that I only read the first paragraph.

For the one request that is cancelled/rejected, but not for every other request behind them in a queue.

It's useful still! If you have an application that talks to another application on the other side of the world, you get to control how fast it sends you stuff with your reading speed. If you stop reading from the socket, the sending application will stop sending before long. It seems like spooky action at a distance but it's just backpressure propagating through many layers.