Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Queues don't fix overload (2014)

ferd.ca

Post reply on HN
Print view View original

41–50 of 156 posts

Re: Queues don't fix overload (2014)

#41

Post by sethammons »

mhandley wrote: post #36
sokoloff wrote: post #33

Earlier quoted context omitted.

Re: your edit: I think in latency insensitive applications, having standing queues that never empty can be totally fine. We custom manufacture ordered items and have priority queues that process input orders into machine-ready output files. We don't need those queues to ever empty as the fastest delivery times are 3 days and the slowest typical is 14 days. Some of the file-prep software is licensed (so we have econom…

Agreed, if there's no latency cost and no cost to maintaining the queue. But if your queue is never emptying, you've hit an equilibrium where you're persistently overloaded and constantly load shedding. In that case you can reduce the queue size so that it is only just not emptying - you'll still be running at 100% capacity and shedding the same amount of demand. And if there was a latency cost or a cost per item in…

> But if your queue is never emptying, you've hit an equilibrium where you're persistently overloaded and constantly load shedding

I believe this is an incorrect assertion. You have hit equilibrium, yes. But you are not load shedding unless you are preventing new queue entrants. And if you are allowing new entrants and the latency is acceptable to customers, you are not overloaded. Like you said, you are at an equilibrium.

Re: Queues don't fix overload (2014)

#42

Post by bhaak »

What's the snarky bit at the end of article about?

> And then of course, there's the use case where you use the queue as a messaging mechanism between front-end threads/processes [...] because your language doesn't support inter-process communications.

Isn't it the other way around? A message broker is used as a simple queue but then you immediately have the option to scale up to n producers and m consumers if demand requires is.

But even if there is only one single producer and one single consumer you still got the advantage of the decoupling of the two sides of the queue.

Re: Queues don't fix overload (2014)

#44

Post by datadrivenangel »

mhandley wrote: post #36
sokoloff wrote: post #33

Earlier quoted context omitted.

Re: your edit: I think in latency insensitive applications, having standing queues that never empty can be totally fine. We custom manufacture ordered items and have priority queues that process input orders into machine-ready output files. We don't need those queues to ever empty as the fastest delivery times are 3 days and the slowest typical is 14 days. Some of the file-prep software is licensed (so we have econom…

Agreed, if there's no latency cost and no cost to maintaining the queue. But if your queue is never emptying, you've hit an equilibrium where you're persistently overloaded and constantly load shedding. In that case you can reduce the queue size so that it is only just not emptying - you'll still be running at 100% capacity and shedding the same amount of demand. And if there was a latency cost or a cost per item in…

It sounds like the above poster is in a situation with multiple priority queues, where as long as latency and throughput for high priority items is good, having higher latency for lower priority items is fine, and could be a form of low-cost back pressure on new orders or signalling to improve the throughput a little bit.

Re: Queues don't fix overload (2014)

#45

Post by 0xbadcafebee »

manuelabeledo wrote: post #34
0xbadcafebee wrote: post #22

Earlier quoted context omitted.

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…

> Nobody seems to talk about this, I guess because it's taken for granted? No, because it has a theoretical limit, same as queues, back pressure, etc. One cannot simply scale up indefinitely because it is not profitable.

There's no product in the world that would hit a limit if autoscaled globally on AWS. Sure, you could write an app whose literal sole purpose is "take up all memory, CPU and bandwidth, recursively, infinitely", but nobody is making that product. Real products built today have a finite amount of demand, and global cloud capacity is larger than that.

You can't say what architecture is or isn't profitable in general, business is more complicated than that. But besides the realities of commerce, you can easily architect a system such that the autoscaling cost is a fraction of opex.

Re: Queues don't fix overload (2014)

#46

Post by secondcoming »

0xbadcafebee wrote: post #22
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…

That's fine until the issue lies with something that your auto-scaled instances talk to, e.g. Redis, Scylla, SQL DB

Re: Queues don't fix overload (2014)

#47

Post by secondcoming »

0xbadcafebee wrote: post #22
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…

That's fine until the issue lies with something that your auto-scaled instances talk to, e.g. Redis, Scylla, SQL DB. There are situations where auto-scaling to infinity makes things far worse.

Re: Queues don't fix overload (2014)

#49

Post by eddd-ddde »

mhandley wrote: post #18

What queues do is smooth out the mismatch between supply and demand. If the mismatch lasts long enough, the queue will overflow, and then you need to load shed (and you need to plan for what the least bad way of load shedding is). But queues do increase overall throughput, up to a point. If the demand was bursty on short timescales and you only allow a small queue to build before load-shedding, you may be wasting cap…

The simplest example, try to empty a bucket on a pipe, you'll have a real struggle, you are gonna be standing there for awhile. Now empty a bucket in a sink, yeah it will fill up and maybe you can do two buckets at once, but after dumping it in one second you can leave it unattended and know that all that water will drain correctly.

Re: Queues don't fix overload (2014)

#50

Post by kqr »

mhandley wrote: post #18

What queues do is smooth out the mismatch between supply and demand. If the mismatch lasts long enough, the queue will overflow, and then you need to load shed (and you need to plan for what the least bad way of load shedding is). But queues do increase overall throughput, up to a point. If the demand was bursty on short timescales and you only allow a small queue to build before load-shedding, you may be wasting cap…

It's important to note that for interactive applications, queues can have surprisingly little capacity and still provide all the demand levelling necessary. Technically this depends on the variance of the arrivals, but in my practical experience, even quite bursty arrivals are often well-behaved enough that a second or so of queue capacity is sufficient.

Any time I've had queues need more than a second of capacity has been when the system is broken and user experience has already degraded to the point where it might be better for the median user to shed a small amount of load to get back to a sub-second queue.

(In batch processing where you aim for maximum throughput, obviously you need a long queue to make sure you never run out of job.)

Post reply on HN