Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Queues don't fix overload (2014)

ferd.ca

Post reply on HN
Print view View original

71–80 of 156 posts

Re: Queues don't fix overload (2014)

#71

Post by PaulHoule »

tonyarkles wrote: post #63
PaulHoule wrote: post #31

Earlier quoted context omitted.

Whenever I talk to operations research people about "how do I learn X?" or "how do I calculate Y?" I usually get told to write a Monte Carlo simulation despite there being a lot of beautiful math involving stochastic processes, generating functions and stuff like that. (Even if you are calculating results in closed form it is still a slam dunk to have a simulation to check the work except when you are dealing with "e…

I learned about this stuff in grad school. The course wasn't mandatory for everyone but my supervisor made it mandatory for me due to the nature of the research I was doing: "Computer Systems and Performance Evaluation". It was basically focused on queuing theory and state space modelling. Reading through this whole discussion thread really makes me want to dig up my old notes and whip up a blog post with a Jupyter n…

If I were you I'd consider using

https://simpy.readthedocs.io/en/latest/

inside a Jupyter notebook.

Re: Queues don't fix overload (2014)

#72

Post by jeremyjh »

0xbadcafebee wrote: post #22
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…

Autoscaling is not going to help if you are IO-bound in your database. One point of the article is you have to identify your bottleneck before you can make sensible design choices.

Re: Queues don't fix overload (2014)

#73

Post by zokier »

What's tricky is that generally queues are difficult to avoid; they are everywhere in computing. Network has queues, hardware has queues, services have queues, operating systems have queues, frameworks and runtimes have queues, your clients will have queues. You often have very limited visibility on many of those, nevermind having any controls, especially if you are not even aware that the queue exists.

Re: Queues don't fix overload (2014)

#74

Post by tonyarkles »

weego wrote: post #4
cnity wrote: post #3

Earlier quoted context omitted.

The sink overflow goes to the same drain, so it doesn't really work in this analogy! (Or it works very well, depending on your perspective).

The overflow goes to the same drain, but further down, past a bottleneck that's designed to be there for a logical reason. It's a good example of real world flow control considerations vs paper theory.

Continuing the analogy from the article though, the sink overflow doesn't go past the red arrow in their diagrams. It's there to solve the specific problem of "the sink drain is plugged, either because you left the water running with the plug in or because there's a ball of hair immediately below the drain".

Re: Queues don't fix overload (2014)

#75

Post by vault_ »

0xbadcafebee wrote: post #22
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…

Autoscaling seems like a downstream concern from the techniques being discussed here. Autoscaling tends to have a pretty high latency, so you still need a strategy for being overloaded while that extra capacity comes online. There's also a question of how the autoscaler knows what "load" is and when it's "too high." Just going off of CPU/memory usage probably means you're over-provisioning. Instead, if you have back-pressure or load-shedding built into your system you can use those as signals to the autoscaler.

Re: Queues don't fix overload (2014)

#76

Post by hinkley »

kqr wrote: post #55
dijit wrote: post #15

Earlier quoted context omitted.

I think it's the threshold of all people to truly understand that there's no solution to certain problems, only tradeoffs. Queues are great, but can lead to catastrophic failure if you don't have a good way of handling the queue, so making an active choice about how you handle overload is part of designing a robust and resilient system. Trading off new requests for current requests is, in my experience, a valid strat…

> Trading off new requests for current requests is, in my experience, a valid strategy for eCommerce for example. We called it "quenching". I'm not sure in which direction the trade happens but it sounds like you're dropping older requests in favour of newer. I agree, this has worked well for me also. Surprisingly often the oldest item in the queue is the one for which service will be least valuable.

I think the theory is that if half of your excess requests come from two actors, then half of the consequences of random dropping are felt by those two people. That’s not fair, but it’s better than dropping all requests after you get into trouble. Fairness is very difficult to achieve in a distributed system, due to coordination delays.

Re: Queues don't fix overload (2014)

#78

Post by hinkley »

0xbadcafebee wrote: post #22
kasey_junk wrote: post #8

This is a weird article because it points out that queues don’t solve overload but neither do load shedding or back pressure. All 3 techniques are just different trade offs on what to do in the face of overload. All 3 have negative ramifications for the users of the system. Load shedding reduces availability, back pressure increases complexity and queues increase latency. In “critical” systems you need all 3. And all…

The only real solution to overload (that is, the eventuality of the system not having enough capacity), in modern systems, is autoscaling. Nobody seems to talk about this, I guess because it's taken for granted? But you can literally just keep adding capacity now. We didn't really have that before the cloud; you had the servers you bought, and maybe you'd rush to repurpose some servers to add capacity. Now an algorit…

If I let my customers railroad me into running more servers to fulfill their “needs” then I may transition into losing money on my business. That’s not a solution.

Needs is in scare quotes because a lot of traffic comes from misunderstanding or laziness from customers or from other divisions. Try as we might, nearly all of the improvements in capacity per customer on my project in the last six months have come from that sort of work. Other sorts of changes have unfortunately been too close to the noise floor. They add up but are difficult to demonstrate, and are very easy to undo with new feature work.

Re: Queues don't fix overload (2014)

#79

Post by klabb3 »

supertrope wrote: post #65

The US Veterans Affairs system has waiting lists for medical care. Due to Congressional oversight the length of this waiting list was scrutinized. Adding capacity through hiring more medical personnel takes more budget and time. Load shedding through not letting people get on the wait list is unacceptable. So the bureaucracy added an additional buffer. They added patients to a secret overflow wait list and moved them…

That’s an innovative way of doing it! Another, much more prevalent way is: only open appointments for 1 week/month ahead, then reject new bookings when full. Tell your clients “just book fast at Monday X am” knowing full well it’s a damn lottery. Or even worse, don’t tell customers about the release time “to fight bots”.

Then of course, blame bots for taking the appointments, and somehow fool people that bots are the reason for the size of the real queue.

This has come up so many times I’ve lost count. Usually with governments or some service that a corporation provides reluctantly. It does one thing really well: obscure the true size of the queue. Just like in the article, that’s a feature.

Re: Queues don't fix overload (2014)

#80

Post by Joker_vD »

User23 wrote: post #68
Joker_vD wrote: post #32

Earlier quoted context omitted.

> If... you can prioritize requests in a timely fashion Just want to point out that this still is processing ― and so your sorting-proxy that would either route high-priority requests forward for actual processing or put low-priority requests into a queue with unbounded storage can still be overloaded with incoming requests. And, of course, that queue with unbounded storage need to be able to put all of those incomin…

Sure, and at some level everything is processing. Even putting an object at the end of a FIFO queue has overhead. I would be very hesitant to introduce a sorting proxy unless the need for it had been thoroughly demonstrated though. If your prioritization isn’t lightweight enough to do in the server it’s probably too heavy. Also you might accomplish it at the network level. Regardless your point stands that the work h…

> I would be very hesitant to introduce a sorting proxy

> fair queuing is a good idea for the majority of production services

Huh? As you've said, fair queuing requires one to quickly look at incoming messages, decide whether they are high/low-priority and route accordingly (to the processing server/to the queue). This is a sorting proxy, and so it seems that having it in some form is required to implement fair queuing. Right?

> It’s absolutely great when a misbehaving client only DoSes themselves and doesn’t affect anyone else.

Yes, of course. But that's precisely the problem with DoS/DDoS attacks: they're, inherently, the cases of a client/clients refusing to obey the back pressure demands from the upstream.

Over network, a misbehaving client affects also affects their ISP at the very least, by the way.

Post reply on HN