Queueing Requests Queues Your Capacity Problems, Too • Hacker News Board

Re: Queueing Requests Queues Your Capacity Problems, Too

#21

Post by mrngm » Fri, Apr 03, 2026, 8:01 PM UTC

mhawthorne wrote: post #18

mrngm wrote: post #2

That reminds me of this talk[0] by Gil Tene called "How NOT to Measure Latency" at the Strangeloop conference in 2015 (or read this blog post[1] that contains the most important points). [0] https://www.youtube.com/watch?v=lJ8ydIuPFeU [1] https://bravenewgeek.com/everything-you-know-about-latency-i...

Author here. That was a great article, thanks for sharing. Especially the part about how your probability of experiencing a p99 latency is much higher than you'd intuit. I don't agree with all of it, but definitely a few points made directly or indirectly hit home, such as: - there is no single metric that can accurately represent "latency" - most of our metrics are misleading in what they unconsciously include or ex…

If you have individual request logs with timing infomation, you could construct that afterwards. It does take some effort to have an effective way of displaying these metrics. Where would you put an individual request that took 532ms and started at t=34.682s? Would you align all requests that started in the 34th second at t=34s, or look at completion time (ie within t=35s)?

Would you rather see "number of requests started at this ms" (you seem to suggest this), or is something else more interesting?

I think a sort of Gantt chart that plots duration of requests as well as starting time within the time span (e.g. a second or more) might be very informative. Each individual request on a different position on the Y axis, time on the X axis. Perhaps you have some bound on requests in flight, that could be the height of the Y axis, so you can easily see calm or busy periods.

At least our observability stack doesn't show this level of detail, but it would be very interesting to have it. (We do have calculated heatmaps based on maximum request time in Grafana, which is at least better than plots of average request times)

Re: Queueing Requests Queues Your Capacity Problems, Too

#22

Post by zamalek » Fri, Apr 03, 2026, 8:20 PM UTC

mhawthorne wrote: post #20

zamalek wrote: post #19

> Here’s an exchange I had on twitter a few months ago: The purple account is just plain wrong. Classically, the full architecture is this (keeping in mind that all rules are sometimes broken): * CQRS is the linchpin. * You generally only queue commands (writes). A few hundreds of ms of latency on those typically won't be noticed by users. * Reads happen from either a read replica or cache. The problem the author fac…

Agree that CQRS seems like a useful way to partitions writes from reads, aka slower requests from faster requests, to avoid many fast requests waiting behind a few slow ones in line. But even if you shifted reads to one or more caches or read replicas, wouldn't those also have queues that will fill up when you are under-provisioned? Note that I'm using the term "queue" pretty loosely, to include things like Redis' ma…

Absolutely. That's typically a good problem to have :). Hopefully you would had gradual enough growth to implement elastic scaling before this is an issue, but you're definitely eventually screwed and have to outright copy what the likes of FAANG do - your startup is a unicorn at that point, so you'd probably already have the talent hired.

Re: Queueing Requests Queues Your Capacity Problems, Too

#23

Post by avidiax » Fri, Apr 03, 2026, 11:22 PM UTC

bluGill wrote: post #15

avidiax wrote: post #7

When I give system design interviews, candidates that start adding queues reflexively to the design always do poorly. Queueing is only useful for a few cases, IMO: * The request is expensive to reject. For example, the inputs to the rejected request also came from expensive requests or operations (like a file upload). So rejecting the request because of load will multiply the load on other parts of the system. You st…

A short queue is different from a long one. Toyota keeps a box of bolts on the line - a type of queue - instead of ordering them individually as needed. However there should be just enough queue - if it backs up that is a problem.

That sort of queue is fundamentally different than network queues. The individual bolts are not impatient waiting to be installed. There is no P99 bolt install latency measuring from when the bolt arrives in the factory.

You therefore only need enough bolts at each station that they won't run out before the restocker completes a lap, and such that there aren't so many that they get in the way.

Re: Queueing Requests Queues Your Capacity Problems, Too

#24

Post by avidiax » Fri, Apr 03, 2026, 11:34 PM UTC

thaumasiotes wrote: post #9

mankyd wrote: post #8

Use a stack? LIFO. As long as you have capacity to keep it mostly empty, it's fine. When requests backup, at least some people will still get quick responses, instead of making everyone suffer.

For a queue, a backup means that every request (from "now" on, until the end of time) is delayed. For a stack, a backup means that some requests are informally forgotten, and although they still appear to be open, they will not complete until the end of time. That's worse. It's a better match to the behavior you want, except for the part where the old requests still appear to be open. You need to actually close them.…

> You might also want to consider how requesting behavior will change when requests are stacked instead of queued. As soon as people have learned that you keep requests in a stack, the correct way to make a request is to make it, wait for a very small amount of time, and then, if your request hasn't already succeeded, repeat it.

It would be very hard to learn this so long as the queue is a very small fraction of the total throughput. If the queue depth is 100, and you receive 10,000qps, but process 9,900 qps, the queue will get full, and roughly 97 calls will go unanswered. Ideally you should have another mechanism to time these out, which most systems do. Whatever queue type you pick, you are going to reject 1% of the inbound, but with a FIFO queue, you will also delay 100% of the responses. Do that at several layers, and you can even end up with the client timing out even though their request wasn't even rejected at any stage.

Re: Queueing Requests Queues Your Capacity Problems, Too

#25

Post by tatersolid » Sat, Apr 04, 2026, 10:00 AM UTC

ajb wrote: post #12

thaumasiotes wrote: post #10

Earlier quoted context omitted.

> An underappreciated queue type is LIFO (last-in, first-out). It sounds unfair, but it keeps you from moving the median response time at the cost of the maximum response time Why is that beneficial?

Suppose you are a building contractor. You have given start dates for future jobs, but your current job is going to run over the expected time. You can choose between: 1 slip every job, annoying all of the customers whose jobs are queued up. You get a bad reputation. 2 Move onto the next job on time, and gradually complete the stalled job in the background by sending workers back to it when you have spare (which you…

I’ve observed airlines will do this as well if they have maintenance or gate queues. They will sacrifice 1-2 flights (hours late or even cancelled) to keep many other flights near on-time. Fewer angry customers, better reported average “on-time” metrics.

Re: Queueing Requests Queues Your Capacity Problems, Too

#26

Post by mhawthorne » Tue, Apr 07, 2026, 8:22 PM UTC

mrngm wrote: post #21

mhawthorne wrote: post #18

Earlier quoted context omitted.

Author here. That was a great article, thanks for sharing. Especially the part about how your probability of experiencing a p99 latency is much higher than you'd intuit. I don't agree with all of it, but definitely a few points made directly or indirectly hit home, such as: - there is no single metric that can accurately represent "latency" - most of our metrics are misleading in what they unconsciously include or ex…

If you have individual request logs with timing infomation, you could construct that afterwards. It does take some effort to have an effective way of displaying these metrics. Where would you put an individual request that took 532ms and started at t=34.682s? Would you align all requests that started in the 34th second at t=34s, or look at completion time (ie within t=35s)? Would you rather see "number of requests st…

Good question - the question of whether to log the millisecond when the request starts or ends is a great example of how complex these things are to think about accurately, let alone capture.

I'd want to log when the requests start, as I'm mostly concerned with how well-distributed request arrival was at that level of granularity.

I wondered if the network layers in between my client and server were effectively "smoothing" request arrival across each second, or if instead requests were very bursty so that a per-minute spike in a typical graph was dominated by a few seconds or milliseconds within that minute.