Things we finally know about network queues (2017)
1–10 of 47 posts
Re: Things we finally know about network queues (2017)
#2This is a good collection of recommendations. At a new job, I often end up making changes to internal queues that (it turns out) conform to these recommendations, and get better performance and reliability out of it.
Though I didn't learn it by engineering networks like the author -- I took the ideas from lean product development and applied them also to software.
> Corollary: limit queue length to a statistically large burst (eg. 99th percentile).
This is somewhat underspecified -- at what time frame are we calling it a burst? How much money/memory/resources are we willing to spend to maintain the queue?
But, critically, are we willing to make a queue so long that we sacrifice throughput during a sustained overload? Sometimes it makes sense to handle only a 20 % burst or even less, because the shorter queues lead to nicer behaviour when the burst draws out into sustained overload.
Re: Things we finally know about network queues (2017)
#3This is a good collection of recommendations. At a new job, I often end up making changes to internal queues that (it turns out) conform to these recommendations, and get better performance and reliability out of it. Though I didn't learn it by engineering networks like the author -- I took the ideas from lean product development and applied them also to software. > Corollary: limit queue length to a statistically la…
Round-trip times provide an upper bound on what should be considered a burst, since one RTT is enough time to provide feedback to throttle a burst. Of course, different traffic flows through a bottleneck can have drastically different RTTs, so this isn't too helpful.
In practice, CoDel works well for almost all current network technologies with its target parameter at the default of 5ms of allowable queueing delay. Given the structure of today's networks, an individual packet is unlikely to pass through more than a handful of congested bottlenecks, and a small multiple of 5ms of added delay is a tolerable worst-case comparable to the speed of light delays of long-distance connections.
It's definitely somewhat unsatisfying to not have formal derivations of optimal parameters. But reasonable defaults that have been tested in the real world are still a huge improvement over the old way of having network devices that don't even attempt to handle congestion intelligently.
Re: Things we finally know about network queues (2017)
#4Why do we know these things? Why are they true? Does anyone have resources for someone who doesn't know that much about networking justifying these statements?
Re: Things we finally know about network queues (2017)
#5Why do we know these things? Why are they true? Does anyone have resources for someone who doesn't know that much about networking justifying these statements?
A lot of that follows from a network-specific application of queuing theory: https://en.wikipedia.org/wiki/Queueing_theory
I'd start there, and branch out to the various links from it or use the keywords you come across to make more searches.
Re: Things we finally know about network queues (2017)
#6I haven't had time to read through the whole text but lots of it rings true with my understanding. I wrote an article that, if you like this one, you will probably also gain from: https://dev.to/rkeene/why-is-there-packet-loss-1p8a
Re: Things we finally know about network queues (2017)
#7Why do we know these things? Why are they true? Does anyone have resources for someone who doesn't know that much about networking justifying these statements?
This article was me trying to write down my internal notes on the topic at the time, after having dived very deeply into problems with the set of interlocking network queues inside embedded wifi devices we were working on.
I was probably not the first to realize that this set of rules probably applies to all kinds of networks, not just ones in embedded device firmware, but I hadn’t really seen it written down before.
But I’m surprised the article ever made it to the HN front page, even 5 years later, since it doesn’t even attempt to address the how/why/how do you know sorts of questions. It’s mainly a placeholder just so I don’t forget. (Those rules for muxes and demuxes and backpressure are really confusing, but I believe them to be strictly correct.)
The rule about bottlenecks (there is only ever one) I borrowed from the TCP BBR paper. The rule about queues on a path always being empty except for exactly one that is always full, I think I borrowed from a talk by Stuart Cheshire that I can never seem to find when I look.
Re: Things we finally know about network queues (2017)
#8Why do we know these things? Why are they true? Does anyone have resources for someone who doesn't know that much about networking justifying these statements?
> Does anyone have resources for someone who doesn't know that much about networking justifying these statements?
This introduction on queues in the linux network stack is pretty neat: http://www.coverfire.com/articles/queueing-in-the-linux-netw...
While apenwarr themselves have written a bit about bandwidth v latency, featuring bufferbloat and fq_codel: https://apenwarr.ca/log/20180808
> Why do we know these things? Why are they true?
Some of the deduced insights can be traced in queueing theory: https://kottke.org/19/01/its-time-for-some-queueing-theory As always, the hardwork is in figuring out the right balance given cause and effect (which are time-consuming, if not hard, to deduce in the first place).
See also: A recent discussion on the topic: https://news.ycombinator.com/item?id=29220338
Re: Things we finally know about network queues (2017)
#9It occurs to me that most of these queue size tradeoffs would be eliminated if the queue operated in a LIFO manner (a stack) instead of FIFO. That way, a burst can naturally get absorbed and re-emitted, but steady state high load doesn't result in increased latency, just packet loss. Can someone with more knowledge of networking enlighten me on why this is a terrible idea?
Re: Things we finally know about network queues (2017)
#10It occurs to me that most of these queue size tradeoffs would be eliminated if the queue operated in a LIFO manner (a stack) instead of FIFO. That way, a burst can naturally get absorbed and re-emitted, but steady state high load doesn't result in increased latency, just packet loss. Can someone with more knowledge of networking enlighten me on why this is a terrible idea?
Wouldn't this cause lots of out-of-order packets?