Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Queues don't fix overload (2014)

ferd.ca

Post reply on HN
Print view View original

101–110 of 156 posts

Re: Queues don't fix overload (2014)

#101

Post by paulddraper »

0xbadcafebee wrote: post #22
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…

While that is a "real" solution, it not necessarily a possible solution.

1. It may be cost prohibitive.

2. Some systems don't scale horizontally. (In particular, databases.)

3. Many systems do not scale linearly.

Re: Queues don't fix overload (2014)

#102

Post by manuelabeledo »

0xbadcafebee wrote: post #45
manuelabeledo wrote: post #34

Earlier quoted context omitted.

> Nobody seems to talk about this, I guess because it's taken for granted? No, because it has a theoretical limit, same as queues, back pressure, etc. One cannot simply scale up indefinitely because it is not profitable.

There's no product in the world that would hit a limit if autoscaled globally on AWS. Sure, you could write an app whose literal sole purpose is "take up all memory, CPU and bandwidth, recursively, infinitely", but nobody is making that product. Real products built today have a finite amount of demand, and global cloud capacity is larger than that. You can't say what architecture is or isn't profitable in general, bu…

> There's no product in the world that would hit a limit if autoscaled globally on AWS.

While this may be true, autoscaling indefinitely is still not profitable.

> You can't say what architecture is or isn't profitable in general, business is more complicated than that. But besides the realities of commerce, you can easily architect a system such that the autoscaling cost is a fraction of opex.

Customer acquisition and retention cost more money than it should.

I cannot count how many times I got an email from the CTO asking to lower our cloud costs, while product and marketing refuse to raise prices, making the whole endeavor borderline unprofitable. You may then argue that the product suffers from either bad architecture, or bad pricing, or both, but the situation is far from uncommon.

Re: Queues don't fix overload (2014)

#104

Post by emn13 »

naasking wrote: post #91
10000truths wrote: post #89

Earlier quoted context omitted.

Maximum acceptable response times for websites is a few seconds. The HN/Slashdot effect lasts for minutes to hours. That time scale is way too long for a queue to be effective.

You're missing the point. We're talking about general systems here, not websites specifically, and the Slashdot effect is a perfect example that everyone is familiar with where queues do solve maintain availability if longer latency is acceptable.

Furthermore, even as a website aiming to survive some momentary spike in performance - slowing everything down for everyone _can_ be part of a solution to serve more people but fewer things each. People might not normally be willing to wait for more than a second or two for a load; but when they expect or have a sign that things are slower than normal, they might have a little more patience (or just come back to that tab later).

I think people are a little too negative on queues. Sure, really naively implemented with entirely unbounded capacity they're an accident waiting to happen... but you don't have to do that.

Re: Queues don't fix overload (2014)

#105

Post by jeremyjh »

0xbadcafebee wrote: post #100
jeremyjh wrote: post #72

Earlier quoted context omitted.

Autoscaling is not going to help if you are IO-bound in your database. One point of the article is you have to identify your bottleneck before you can make sensible design choices.

....you can autoscale database read replicas, and write nodes (master-master)....

If you are bound in disk IO, adding master-master write nodes will not help you. The same number of bytes have to be written to the same volume whether they come from a replica or an application server. The only solution is partitioning/sharding, and there is no "easy button" to press and make that happen because it comes with its own limitations and trade-offs, and is something the application code will be intimately aware of.

Re: Queues don't fix overload (2014)

#106

Post by dtaht »

Really good thread. Several comments:

Flow Queuing allows applications not creating queues to bypass those that are. It is mildly different from fair queuing: https://ieeexplore.ieee.org/document/8469111

Load shedding, at least for packets, benefits from head drop more than tail drop. Only the codel algorithm does this but codel has been applied to other forms of queuing like Uber dealing with a concertgoer overload.

Kathie Nichols has given some great presentations lately: https://www.understandinglatency.com/

There are a lot of unbounded queues in many applications & libraries, even in rust. They make me twitchy. Even with bounded queues the number chosen is usually arbitrary. I wish a timed queue was a default rather than a length.

I highly recommend Kleinrocks work on these subjects.

I am grumpy big vendors like juniper have yet to adopt smart queues... despite the obvious benefits. https://blog.cerowrt.org/post/juniper/

Re: Queues don't fix overload (2014)

#107

Post by lanstin »

o11c wrote: post #97
hinkley wrote: post #78

Earlier quoted context omitted.

If I let my customers railroad me into running more servers to fulfill their “needs” then I may transition into losing money on my business. That’s not a solution. Needs is in scare quotes because a lot of traffic comes from misunderstanding or laziness from customers or from other divisions. Try as we might, nearly all of the improvements in capacity per customer on my project in the last six months have come from t…

Additionally, most users actually are happy with "slow me down if I'm making too many requests". This is much simpler than recovering from errors correctly.

Which is why your test integration environment has to throw 429 and 5xx errors consistently from day one. The error handling paths are hard to retrofit but easy to do before deployment.

Re: Queues don't fix overload (2014)

#108

Post by lanstin »

0xbadcafebee wrote: post #45
manuelabeledo wrote: post #34

Earlier quoted context omitted.

> Nobody seems to talk about this, I guess because it's taken for granted? No, because it has a theoretical limit, same as queues, back pressure, etc. One cannot simply scale up indefinitely because it is not profitable.

There's no product in the world that would hit a limit if autoscaled globally on AWS. Sure, you could write an app whose literal sole purpose is "take up all memory, CPU and bandwidth, recursively, infinitely", but nobody is making that product. Real products built today have a finite amount of demand, and global cloud capacity is larger than that. You can't say what architecture is or isn't profitable in general, bu…

I have made and deployed pieces of that infinite recursive. Most spectacularly by having infinite call loops triggered. Had we had autoscaling rather than sharp queue limits leading to load shedding it would have been worse.

Re: Queues don't fix overload (2014)

#109

Post by adrianmonk »

> All of a sudden, the buffers, queues, whatever, can't deal with it anymore. You're in a critical state where you can see smoke rising from your servers, or if in the cloud, things are as bad as usual, but more!

There's a valid point here, which is that queues can mask problems. Everything seems fine for a while. Until suddenly it isn't.

Queues take away important feedback about load. Without feedback, you don't know that problems are looming. You also may falsely believe you've fixed something when you really haven't.

But, there's a solution: monitoring and alerting on the queue. Don't alert when it is just about full or just about as bad as you can allow. Alert as soon as it starts to grow beyond normal.

Some possible metrics:

(1) Number of enqueued items exceeds some threshold. Simple, but you may have to readjust the threshold from time to time. (And if the threshold is too low, people may learn to ignore this alert.)

(2) Length of time the most recently dequeued item had been sitting in the queue. When you dequeue an item, if it's been in there for hours (and you were expecting minutes), something is probably wrong.

(3) How long it has been since the queue was (within epsilon of) empty. If you kinda mostly need items to be processed immediately and the queue is a fallback, it shouldn't be used very often or for long stretches of time. (You could also alert on what percentage of the time, over some time window, it was / wasn't empty.)

(4) How long it has been since a worker (successfully) processed an item taken from the queue. If all your workers die, you might as well know about it right now. (You need to somehow account for empty queues, i.e. workers not doing anything because there's no work to do.)

Re: Queues don't fix overload (2014)

#110

Post by lanstin »

jeremyjh wrote: post #105
0xbadcafebee wrote: post #100

Earlier quoted context omitted.

....you can autoscale database read replicas, and write nodes (master-master)....

If you are bound in disk IO, adding master-master write nodes will not help you. The same number of bytes have to be written to the same volume whether they come from a replica or an application server. The only solution is partitioning/sharding, and there is no "easy button" to press and make that happen because it comes with its own limitations and trade-offs, and is something the application code will be intimatel…

Auto sharding. A nice idea for a DAO layer.
Post reply on HN