Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Queues don't fix overload (2014)

ferd.ca

Post reply on HN
Print view View original

111–120 of 156 posts

Re: Queues don't fix overload (2014)

#111

Post by lanstin »

tonyarkles wrote: post #59
secondcoming wrote: post #47

Earlier quoted context omitted.

That's fine until the issue lies with something that your auto-scaled instances talk to, e.g. Redis, Scylla, SQL DB. There are situations where auto-scaling to infinity makes things far worse.

100% that was my first thought when reading the GP comment. Autoscaling isn't a magic fix. Just like the article says, you need to find the red arrow first to figure out where the bottleneck is and whether or not that bottleneck is actually something within the auto-scaling context or not. You've pointed out a couple of good examples of potential bottlenecks. Another possibility is a downstream 3rd-party service. If…

Priority queues for your downstream calls to rate limit them. A large enough cluster can take down most any data store, except maybe spanner. Even then you could increase latency a lot while it scales up.

Re: Queues don't fix overload (2014)

#112

Post by lanstin »

epr wrote: post #93
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

Pretty ironic, right? The entire article is the process of the author making the same mistake he's complaining about. For anyone reading the comments looking for more helpful guidance, the most generally helpful advice you will get is to start by measuring. Your measurements should help you identify bottlenecks, but the answer of how to fix the issue is dependent on your problem's constraints. In fact, there's such a…

Your system has a capacity and limits. They can either be explicitly configured in with deliberate load shedding behavior or implicit with a devil may care behavior. Some where between not getting dial tone when you pick up the phone to weird vocal distortions when the limits are hit, choose your poison. Or let randomness choose.

Re: Queues don't fix overload (2014)

#113

Post by lanstin »

marcosdumay wrote: post #35
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

No, load shedding and back pressure present you trade-offs to deal with an overloaded system. Queues don't. Queues just present you problems. If you take an overloaded system and add a queue, every single feature either gets worse or doesn't get any better. And people like to deny this, and pretend that queues will help. They absolutely help with a lot of things but they do nothing but harm in front of an overloaded…

A bounded queue that quickly errors back to the caller when full is a fine load shedding system. Unbounded queues are death.

Re: Queues don't fix overload (2014)

#114

Post by lanstin »

marcosdumay wrote: post #95
naasking wrote: post #51

Earlier quoted context omitted.

Your mistake is characterising all overloads as the same. Adding a queue to a system that is consistently overloaded won't solve the overload, but many overloads are temporary/bursty, like the Slashdot/HN effect. Queues absolutely do solve these kinds of overloads simply by increasing latency, assuming increased latency is an acceptable choice in your context, of course.

After the overload fixes itself somehow, a queue absolutely solves the availability problem. Yep. If the overload doesn't last for too long. What isn't happening on your example is a queue improving anything on a overloaded system. Queues are immensely helpful for all kinds of problems. Just not for overload.

An unbounded queue means you are allocating a lot of stuff during the time you want to cap end all resources handling traffic. It means you slow down when busier. Depending on the framework it often means the system can't recover automatically but will crash if the queue is in memory or need DBAs to flush the queues or whatever.

Re: Queues don't fix overload (2014)

#115

Post by Sohcahtoa82 »

User23 wrote: post #6

So far as I know there is no theoretical alternative to load shedding or increasing handling capacity if your average request arrival rate is greater than your average request handling rate. At least, not if you want to handle every accepted request using a finite queue[1]. It would appear that with an unbounded queue every request will eventually be handled, but with an unbounded latency guarantee. Which appears equ…

> However, that may very well change with fair queueing.

It doesn't matter what kind of queueing system you put in place. No matter how fancy it is, and even if the queue adds negligible overhead, if your average request arrival rate is higher than the average handling rate, your queue will grow unbounded. Then, for some requests, the system will appear to grind to a halt.

There's no way around it. A priority queue just means that high-priority requests get processed faster. Low-priority requests might end up never getting handled.

Here's a concrete example. I've got a web service that requires a login. I'm using bcrypt to hash passwords with a sufficiently large work factor that I can only hash 5 passwords per second and have a queue for the password hashing. If 6 people are trying to login every second, then after 1 minute, I will have received 360 login requests. 300 of them will have completed, but the queue has grown to 60. Anybody trying to log in at that point will have to wait a minimum of 12 seconds to get their login processed.

But what if I also have users trying to sign up? I have to hash their password, so those get added to the queue as well. If I prioritize logins over signups, then under my scenario, a signup will never complete.

There's really nothing that can be done. An unbounded queue means unbounded latency. Eventually, a login request would have to get rejected just because the queue is full. Of course, in the real world, this creates frustration for the user. They'll either try to login again, or give up.

Re: Queues don't fix overload (2014)

#116

Post by lanstin »

emn13 wrote: post #104
naasking wrote: post #91

Earlier quoted context omitted.

You're missing the point. We're talking about general systems here, not websites specifically, and the Slashdot effect is a perfect example that everyone is familiar with where queues do solve maintain availability if longer latency is acceptable.

Furthermore, even as a website aiming to survive some momentary spike in performance - slowing everything down for everyone _can_ be part of a solution to serve more people but fewer things each. People might not normally be willing to wait for more than a second or two for a load; but when they expect or have a sign that things are slower than normal, they might have a little more patience (or just come back to that…

The distinction is really not queues or not but unbounded in theory or not. There are queues everywhere, connection pools, unparsed messages, packets in flight, memory for calls sent out and not yet replied or timed out, etc. but one unbounded (in theory; they are never unbounded in practice) queue can make your system go from robust to fragile.

Re: Queues don't fix overload (2014)

#117

Post by lanstin »

robertlagrant wrote: post #14
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

> Frankly, your load shedding or back pressure system is probably implemented on a queue one layer down the abstraction. If you're building an API, you don't care how your clients cope with your backpressure. You just want to avoid overloading your services, and queues will indeed not help you with that past a certain point, whereas backpressure probably will. Or at least you can scale backpressure much more cheaply…

Teach your clients to handle 429 with gradual back off/retries and it works well. Even teach your web UI to degrade under partial failure of widget loads.

Re: Queues don't fix overload (2014)

#118

Post by lanstin »

adrianmonk wrote: post #109

> All of a sudden, the buffers, queues, whatever, can't deal with it anymore. You're in a critical state where you can see smoke rising from your servers, or if in the cloud, things are as bad as usual, but more! There's a valid point here, which is that queues can mask problems. Everything seems fine for a while. Until suddenly it isn't. Queues take away important feedback about load. Without feedback, you don't kno…

At AOL there was a weekly queue depth review meeting that led to hardware ordering requests. Worked great till the business stopped growing.

Re: Queues don't fix overload (2014)

#119

Post by lanstin »

domano wrote: post #12
roenxi wrote: post #7

The other thing to bear in mind about queues is that once they start showing of symptoms of something being wrong, collapse might be just around the corner or it might not be depending on the nature of the load. When congestion spikes start showing it is helpful to know some queuing theory to estimate how close the situation is to eating someone's weekend. Congestion collapses are an interesting time because most peo…

Hey, can you recommend something one might read to get up to speed on queuing theory? I certainly am not aware of it, but work with queues.

There are a billion books on abstract or basic queueing theory (worth reading) but a really good modern paper on distributed software implications is https://pure.psu.edu/en/publications/metastable-failures-in-...

Re: Queues don't fix overload (2014)

#120

Post by citrin_ru »

pjdesno wrote: post #84
PaulHoule wrote: post #40

Earlier quoted context omitted.

See https://en.wikipedia.org/wiki/Bufferbloat I lived in Germany in 1999 and then the internet connection from Germany to the US would get overloaded during the day. At maybe 9am the latency would be then it would start dropping packets. I don't know if it was the intention but it was about as good as a ban on VoIP at preventing people from making international VoIP calls. Today there is more consciousness about the…

See RFC 970, "On Packet Switches With Infinite Storage" by John Nagle: https://datatracker.ietf.org/doc/html/rfc970 Back then (I was studying networking as an undergrad at the time, and interned with the Arpanet team) people really did think of network congestion as a buffer allocation problem, so the obvious solution was more buffering - i.e. adding queues. Nagel was one of the first people to point out the problem…

Sometimes it is - TCP incast [1] in a fast LAN can be mostly alleviated by using switches with large buffers. Generally the higher throughput the bigger buffers you need unless having low latency is more important than low packet loss. The queue size is a tradeoff (as almost everything).

[1] https://www.usenix.org/system/files/login/articles/chen12-06...

Post reply on HN