Why disaster happens at the edges: An introduction to queue theory • Hacker News Board

Re: Why disaster happens at the edges: An introduction to queue theory

#21

Post by tharkun__ » Mon, Nov 15, 2021, 1:10 AM UTC

roenxi wrote: post #14

Queue theory is almost a distraction because it overcomplicates the situation. The underlying insight is that rates matter. If a system can handle 100 events a second, then 99 events per second everything is fine and 101 events per second the system is down. There is a threshold where everything falls apart. Queue theory encourages people to think in terms of flow rates and has the technically correct de-rates to nom…

Why is 101 events per second == system down?

Of course there are systems where not being able to process everything right when it comes in means that you're "down". It's not a given though and queuing systems are actually perfect for use cases where not being able to process at the speed of incoming requests is completely fine. Eventual consistency is a thing.

I have the same experience though that it seems to be hard for folks to accept that yes, 100k messages in _that_ queue is totally fine. Some process dumped 100k messages to then be processed over the next hour and that's totally fine. That way we don't have to actually spend a small fortune to process them at the speed at which they can be generated.

Of course it depends on the problem at hand. If that particular flow in the system is one that is highly user interactive, then yes, this means 'system down' for your users. Never mind that the system will eventually process it all. I always smile when I see another system than ours and I can spot distributed processing w/ queues in between in how the system behaves in certain circumstances.

Re: Why disaster happens at the edges: An introduction to queue theory

#22

Post by mohanmca » Mon, Nov 15, 2021, 2:19 AM UTC

joshuanapoli wrote: post #17

roenxi wrote: post #14

Queue theory is almost a distraction because it overcomplicates the situation. The underlying insight is that rates matter. If a system can handle 100 events a second, then 99 events per second everything is fine and 101 events per second the system is down. There is a threshold where everything falls apart. Queue theory encourages people to think in terms of flow rates and has the technically correct de-rates to nom…

> The battle in an organization is convincing people to look at flow at all. I guess that's the point... an organization needs enough people to know about queue theory "to look at flow" and understand that queues are ubiquitous and anticipate that these queues experience a latency phase change at some point depending on utilization (and job duration variance). If an organization doesn't have anyone who understands qu…

Thanks! I observed two points.

In order to look at flow - "should learn about queue theory"!

"queue theory" itself would be useful to design queuing related systems.

Re: Why disaster happens at the edges: An introduction to queue theory

#23

Post by taneq » Mon, Nov 15, 2021, 2:23 AM UTC

roenxi wrote: post #20

kccqzy wrote: post #15

Earlier quoted context omitted.

> The underlying insight is that rates matter. If a system can handle 100 events a second, then 99 events per second everything is fine and 101 events per second the system is down. Not really. You need actual theory to inform you whether 99 events per second is okay. See https://www.johndcook.com/blog/2008/10/21/what-happens-when-...

If you want to maintain 99.99989% uptime or are about to spend $10 million, yes. If you want the cheap option then measure (arrival rate / service rate) and try to keep that below 80%. Then if that doesn't work adjust accordingly. That isn't really queue theory because it doesn't consider variance and requires effectively no knowledge of literature, what rho means or why people use M/M/1 queues as a default. Don't ha…

There's another advantage to feeling it out in production - you're measuring the actual system, not your theoretical model of the system, so all the extra things you've forgotten are automatically included (assuming you measure the right things, of course!)

And keeping your safety margins nice and wide isn't just inefficiency. It's also building in resilience for when something unexpected does happen. Yes, if you need your six-nines performance on a 1% margin then maybe you should try to optimize this to the nth degree, but realistically almost none of us have to sail that close to the wind.

Re: Why disaster happens at the edges: An introduction to queue theory

#24

Post by layer8 » Mon, Nov 15, 2021, 2:28 AM UTC

jph wrote: post #7

Queueing theory-- my notes for programmers: https://github.com/joelparkerhenderson/queueing-theory Queueing theory is the mathematical study of waiting lines, or queues. We use queueing theory in our software development, for purposes such as project management kanban boards, inter-process communication message queues, and devops continuous deployment pipelines.

The notes don’t explain what problems queue theory solves, i.e. why would I want to use it, as a programmer. The Wikipedia article’s second sentence is a bit more helpful in that respect: "A queueing model is constructed so that queue lengths and waiting time can be predicted."

Re: Why disaster happens at the edges: An introduction to queue theory

#25

Post by dinosaurdynasty » Mon, Nov 15, 2021, 3:20 AM UTC

tharkun__ wrote: post #21

roenxi wrote: post #14

Queue theory is almost a distraction because it overcomplicates the situation. The underlying insight is that rates matter. If a system can handle 100 events a second, then 99 events per second everything is fine and 101 events per second the system is down. There is a threshold where everything falls apart. Queue theory encourages people to think in terms of flow rates and has the technically correct de-rates to nom…

Why is 101 events per second == system down? Of course there are systems where not being able to process everything right when it comes in means that you're "down". It's not a given though and queuing systems are actually perfect for use cases where not being able to process at the speed of incoming requests is completely fine. Eventual consistency is a thing. I have the same experience though that it seems to be har…

If you can only handle 100 events/second and you are getting 101 events/second continuously the queue depth is going to go up by 1 event/second up to infinity, causing your latency to go to infinity.

Re: Why disaster happens at the edges: An introduction to queue theory

#26

Post by jimkleiber » Mon, Nov 15, 2021, 3:20 AM UTC

roenxi wrote: post #14

Queue theory is almost a distraction because it overcomplicates the situation. The underlying insight is that rates matter. If a system can handle 100 events a second, then 99 events per second everything is fine and 101 events per second the system is down. There is a threshold where everything falls apart. Queue theory encourages people to think in terms of flow rates and has the technically correct de-rates to nom…

> The battle in an organisation is convincing people to look at flows at all. In my experience people love metrics that track stock (we have X widgets or can handle Y orders) and not flows (we built A widgets and sold B widgets).

This reminds me of an entrepreneurship class I took in college where the professor emphasized the importance of cash flow, something I hadn't considered much and found somewhat counterintuitive.

I'd love to learn more about this perspective on when to focus on the flows over stocks, are there any resources you'd recommend to help me better understand it?

Re: Why disaster happens at the edges: An introduction to queue theory

#27

Post by NegativeLatency » Mon, Nov 15, 2021, 3:38 AM UTC

tharkun__ wrote: post #21

roenxi wrote: post #14

Queue theory is almost a distraction because it overcomplicates the situation. The underlying insight is that rates matter. If a system can handle 100 events a second, then 99 events per second everything is fine and 101 events per second the system is down. There is a threshold where everything falls apart. Queue theory encourages people to think in terms of flow rates and has the technically correct de-rates to nom…

Why is 101 events per second == system down? Of course there are systems where not being able to process everything right when it comes in means that you're "down". It's not a given though and queuing systems are actually perfect for use cases where not being able to process at the speed of incoming requests is completely fine. Eventual consistency is a thing. I have the same experience though that it seems to be har…

Assuming you have queues and stuff that can hold the requests in the intermediary time. Some systems will just start dropping the data or crashing and loose data.

If you have a queue or something in the middle then the failure is potentially recoverable or never happens in the first place.

Re: Why disaster happens at the edges: An introduction to queue theory

#28

Post by gizmo686 » Mon, Nov 15, 2021, 3:59 AM UTC

dinosaurdynasty wrote: post #25

tharkun__ wrote: post #21

Earlier quoted context omitted.

Why is 101 events per second == system down? Of course there are systems where not being able to process everything right when it comes in means that you're "down". It's not a given though and queuing systems are actually perfect for use cases where not being able to process at the speed of incoming requests is completely fine. Eventual consistency is a thing. I have the same experience though that it seems to be har…

If you can only handle 100 events/second and you are getting 101 events/second continuously the queue depth is going to go up by 1 event/second up to infinity, causing your latency to go to infinity.

The moral of that story is don't let your queue go up to infinity. Drop 1/101 events and serve the other 99% with a typical latency. Instead of latency going to infinity to everyone, your service gracefully-ish degrades by serving as many requests as it is able to and quickly reporting an error to those it is not. Once load drops below capacity, you are immedietly back to a fully functioning state.

Granted, depending on how well behaved your clients are, the number of events each second may as the failed requests are retried.

Re: Why disaster happens at the edges: An introduction to queue theory

#29

Post by btilly » Mon, Nov 15, 2021, 4:02 AM UTC

gizmo686 wrote: post #28

dinosaurdynasty wrote: post #25

Earlier quoted context omitted.

If you can only handle 100 events/second and you are getting 101 events/second continuously the queue depth is going to go up by 1 event/second up to infinity, causing your latency to go to infinity.

The moral of that story is don't let your queue go up to infinity. Drop 1/101 events and serve the other 99% with a typical latency. Instead of latency going to infinity to everyone, your service gracefully-ish degrades by serving as many requests as it is able to and quickly reporting an error to those it is not. Once load drops below capacity, you are immedietly back to a fully functioning state. Granted, depending…

So we should solve the problem at the port of Long Beach by randomly dropping cargo into the ocean until we no longer get container ships piling up off the coast?

Re: Why disaster happens at the edges: An introduction to queue theory

#30

Post by vishnugupta » Mon, Nov 15, 2021, 4:16 AM UTC

btilly wrote: post #29

gizmo686 wrote: post #28

Earlier quoted context omitted.

The moral of that story is don't let your queue go up to infinity. Drop 1/101 events and serve the other 99% with a typical latency. Instead of latency going to infinity to everyone, your service gracefully-ish degrades by serving as many requests as it is able to and quickly reporting an error to those it is not. Once load drops below capacity, you are immedietly back to a fully functioning state. Granted, depending…

So we should solve the problem at the port of Long Beach by randomly dropping cargo into the ocean until we no longer get container ships piling up off the coast?

Or build sufficient back pressure mechanisms so that the cargo doesn’t get loaded in the first place or better still the goods doesn’t get manufactured and so on.

Send the feedback as far up the source as possible.