Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Queues don't fix overload (2014)

ferd.ca

Post reply on HN
Print view View original

141–150 of 156 posts

Re: Queues don't fix overload (2014)

#141

Post by jiggawatts »

0xbadcafebee wrote: post #22
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…

In my experience, auto scaling has not yet been applicable in the typical sense of adding more web servers.

If you use an efficient language like C# or Go, the performance bottleneck moves to the database layer almost immediately.

Caching can be added and can auto scale, but that has a tradeoff: stale data or eventual consistency.

Databases are hard to scale because typically there has to be a single master instance responsible for write ordering and transaction consistency.

Auto scaling the database layer is fraught with its own issues, and simply doesn’t work in practice for many common scenarios.

For typical business apps, or typical web pages that might suddenly get a huge surge in usage (I.e.: Hug of death), a CDN can help… maybe.

My customers keep asking for auto scale to solve their problems, when the root cause is invariably a missing index in their database.

Re: Queues don't fix overload (2014)

#142

Post by aidenn0 »

Corollary: The part of the system with the least amount of queuing will tend to fail the soonest in a situation of overload, thus you can always move the blame to a different part of the system by increasing queuing at the part that is failing.

Re: Queues don't fix overload (2014)

#143

Post by tanveergill »

Animats wrote: post #135

FIFO queues do not fix overload. Fair queuing queues, though, can fix it if the problem is coming from a specific source. I have a web site set up that way. An in-memory table in MySQL is used to handle queuing for a slow operation. The queuing system has some fairness. This works well enough that one site sent bogus requests for a month at a high rate, and it had no effect on actual users. Fair queuing in SQL: SELEC…

I agree that fair queueing is a good solution to this problem! A request scheduler based on weighted fair queuing is also central to Aperture, an open-source load management system my team has been building for the last 2 years. Priorities and tokens (request weights) can be provided to Aperture when scheduling requests. It runs weighted fair queuing that ensures relative allocation of capacity based on the relative values of (token/priorities). It also ensures fair allocation of capacity across users within the same priority class so that no single user can starve others.

Would love to hear your thoughts about this project: https://github.com/fluxninja/aperture

Re: Queues don't fix overload (2014)

#144

Post by tanveergill »

Sohcahtoa82 wrote: post #115
User23 wrote: post #6

So far as I know there is no theoretical alternative to load shedding or increasing handling capacity if your average request arrival rate is greater than your average request handling rate. At least, not if you want to handle every accepted request using a finite queue[1]. It would appear that with an unbounded queue every request will eventually be handled, but with an unbounded latency guarantee. Which appears equ…

> However, that may very well change with fair queueing. It doesn't matter what kind of queueing system you put in place. No matter how fancy it is, and even if the queue adds negligible overhead, if your average request arrival rate is higher than the average handling rate, your queue will grow unbounded. Then, for some requests, the system will appear to grind to a halt. There's no way around it. A priority queue j…

Aperture (https://github.com/fluxninja/aperture) takes a slightly opinionated take on queue size. Instead of defining queue sizes, you put a timeout on each request which is the amount of time the request is willing to wait in the queue. Then we run a weighted fair queuing algorithm that ensures relative allocation across workloads, e.g. 90% capacity for critical requests. But the capacity is allowed to burst when there is low demand.

Re: Queues don't fix overload (2014)

#145

Post by tanveergill »

User23 wrote: post #6

So far as I know there is no theoretical alternative to load shedding or increasing handling capacity if your average request arrival rate is greater than your average request handling rate. At least, not if you want to handle every accepted request using a finite queue[1]. It would appear that with an unbounded queue every request will eventually be handled, but with an unbounded latency guarantee. Which appears equ…

I am in the same school of thought as you, fair queuing is the most optimal solution while capacity is constrained. This is the approach we took in Aperture, an open-source load management system. Read more about our Scheduler which implements a variant of WFQ (Weighted Fair Queuing) to ensure desired capacity allocation across workloads (group of requests at the same priority level) and SWFQ (Stochastic Weighted Fair Queuing) to ensure fairness across users within each workload: https://docs.fluxninja.com/concepts/scheduler

Re: Queues don't fix overload (2014)

#146

Post by tanveergill »

marcosdumay wrote: post #35
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

No, load shedding and back pressure present you trade-offs to deal with an overloaded system. Queues don't. Queues just present you problems. If you take an overloaded system and add a queue, every single feature either gets worse or doesn't get any better. And people like to deny this, and pretend that queues will help. They absolutely help with a lot of things but they do nothing but harm in front of an overloaded…

I agree that queues can cause problems especially when misconfigured. But some amount of queuing is necessary, to absorb short spikes in demand vs capacity. Also, queues can be helpful to re-order requests based on criticality which won't be possible with zero queue size - in which case we have to immediately drop a request or admit it without considering it's priority.

I think it is beneficial to re-think how we tune queues. Instead of setting a queue size, we should be tuning the max permissible latency in the queue which is what a request timeout actually is. That way, you stay within the acceptable response time SLA while keeping only the serve-able requests in the queue.

Aperture, an open-source load management platform took this approach. Each request specifies a timeout for which it is willing to stay in the queue. And weighted fair queuing scheduler then allocates the capacity (a request quota or max number of in-flight requests) based on the priority and tokens (request heaviness) of each request.

Read more about the WFQ scheduler in Aperture: https://docs.fluxninja.com/concepts/scheduler

Link to Aperture's GitHub: https://github.com/fluxninja/aperture

Would love to hear your thoughts on our approach!

Re: Queues don't fix overload (2014)

#147

Post by hinkley »

lanstin wrote: post #107
o11c wrote: post #97

Earlier quoted context omitted.

Additionally, most users actually are happy with "slow me down if I'm making too many requests". This is much simpler than recovering from errors correctly.

Which is why your test integration environment has to throw 429 and 5xx errors consistently from day one. The error handling paths are hard to retrofit but easy to do before deployment.

We have some people who are convinced that Google is punishing us for 429 responses. If that's true, then it sucks the big one.

Re: Queues don't fix overload (2014)

#148

Post by sgarland »

0xbadcafebee wrote: post #100
jeremyjh wrote: post #72

Earlier quoted context omitted.

Autoscaling is not going to help if you are IO-bound in your database. One point of the article is you have to identify your bottleneck before you can make sensible design choices.

....you can autoscale database read replicas, and write nodes (master-master)....

> master-master

Now you have two problems.

Re: Queues don't fix overload (2014)

#149

Post by devdiary »

And this understanding is extremely important when you are working on rate limiting. Instead of controlling the rate of requests, one must control the number of concurrent requests - https://docs.fluxninja.com/concepts/concurrency-limiter

Throughput (number of requests processed) is different from capacity (number of requests that can be handled). Managing capacity is more practical and optimum solution than managing the throughput

Re: Queues don't fix overload (2014)

#150

Post by withinboredom »

aidenn0 wrote: post #140
pjdesno wrote: post #84

Earlier quoted context omitted.

See RFC 970, "On Packet Switches With Infinite Storage" by John Nagle: https://datatracker.ietf.org/doc/html/rfc970 Back then (I was studying networking as an undergrad at the time, and interned with the Arpanet team) people really did think of network congestion as a buffer allocation problem, so the obvious solution was more buffering - i.e. adding queues. Nagel was one of the first people to point out the problem…

Ugh. WiFi signals aren't the best in my A/V cabinet, so I ran G.hn powerline to it. This works great 98% of the time, but occasionally there will be something that blocks traffic for a short period of time, and those things must have huge buffers. If I'm e.g. watching Netflix when there is a hiccup, I see ping times of over a minute! I wrote a program that monitors ping times and reboots the G.hn adapters when they g…

Do you live near an airport (https://en.wikipedia.org/wiki/Dynamic_frequency_selection?

You are probably using channels that interfere with radar and when a router detects the radar, it should shutdown for a minute or so.

Post reply on HN