Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we finally know about network queues (2017)

apenwarr.ca

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Things we finally know about network queues (2017)

#21

Post by kloch »

Excessively large buffers confuses TCP windowing. On network devices like switches and routers buffers are only actually needed for two things: internal switching/routing latency (which is usually constant and very small) and serialization onto output interfaces (which varies by input/output port speed mismatch and/or number of inputs multiplexed to one output). Handling network "burstiness" should be left up to the endpoint TCP stack or applications in the case of UDP. Ethernet flow control should be disabled unless you find it actually helps (unlikely).

On servers buffers have more work to do since the CPU may be busy with many other tasks. It is rare that I have had to adjust the default buffer sizes but keep in mind server buffers too large can increase jitter as well as confuse TCP windowing (depending on when ACK's are sent). Some applications may prefer that packets be dropped instead of queued for excessively long times.

The most important thing is to ensure that selective ACK's and window scaling TCP options are enabled. They are enabled by default in modern operating systems but you might be surprised how often they are disabled by clueless sysadmins. A common cause of this is/was buggy TCP offloading drivers where the apparent "solution" was to disable ALL tcp options instead of just TCP offload. Window scaling in particular is essential with modern port speeds.

Re: Things we finally know about network queues (2017)

#22

Post by Bluecobra »

iforgotpassword wrote: post #15

> Drop packets when demuxing. From y to o, it's best to drop packets at o rather than use backpressure from o to y. Otherwise, a single full output queue could starve the others by stopping y. (Example: imagine if a full queue to a 1 Mbps wifi station could stop traffic to a 100 Mbps wifi station. Some Linux wifi drivers suffer from this.) Aaah, memories. Combined with point 11, pause frames. I was debugging a weird…

Funnily enough there’s another excellent article by apenwarr that touches on this a bit. Ethernet is designed to go as fast as possible/line rate and assumes that everything is the same speed. When you disabled pause frames, it likely prevented the switch from interfering and punting the issue to the TCP stack of the server/client.

https://apenwarr.ca/log/20170810

Re: Things we finally know about network queues (2017)

#23

Post by Bluecobra »

rkeene2 wrote: post #6

I haven't had time to read through the whole text but lots of it rings true with my understanding. I wrote an article that, if you like this one, you will probably also gain from: https://dev.to/rkeene/why-is-there-packet-loss-1p8a

Why wasn’t the speed of the WAN circuit questioned? To paraphrase Google, OC-12 is so slow I forgot how to count that low. If your link is fast enough, you don’t need to worry about QoS, the switch buffer size, or have to buy an expensive WAN traffic shaping device. :)

Re: Things we finally know about network queues (2017)

#24

Post by dtaht »

I tried to leave people laughing, here: https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-...

And the online book, freely available and primarily on applying fq_codel to everything (and also sch_cake) is here: https://bufferbloat-and-beyond.net/

In the last decade we've managed to eliminate fifos from most of linux, most 3rd party firmwares notably with openwrt and sqm, ios and OSX. The only major things left unfixed are unfortunately home routers and edges.

Re: Things we finally know about network queues (2017)

#25

Post by dtaht »

iforgotpassword wrote: post #15

> Drop packets when demuxing. From y to o, it's best to drop packets at o rather than use backpressure from o to y. Otherwise, a single full output queue could starve the others by stopping y. (Example: imagine if a full queue to a 1 Mbps wifi station could stop traffic to a 100 Mbps wifi station. Some Linux wifi drivers suffer from this.) Aaah, memories. Combined with point 11, pause frames. I was debugging a weird…

The modern AQM's are based on "Time in Queue". Both pie and codel work brilliantly with pause frames, so long as BQL is also in the driver. fq+aqm (be it codel, pie or cobalt) works even better.

See:

https://datatracker.ietf.org/doc/html/rfc8290

https://datatracker.ietf.org/group/aqm/documents/

https://arxiv.org/abs/1804.07617

Adding AQM and FQ wifi was way, way harder, (apenwarr drove the group at google that did some of it), but there is full support for fq_codel now in the mt76, ath9k, ath10k, iwl, and one new realtek chipset in the linux kernel. https://lwn.net/Articles/705884/

Re: Things we finally know about network queues (2017)

#26

Post by dtaht »

apenwarr wrote: post #7
moonchild wrote: post #4

Why do we know these things? Why are they true? Does anyone have resources for someone who doesn't know that much about networking justifying these statements?

This article was me trying to write down my internal notes on the topic at the time, after having dived very deeply into problems with the set of interlocking network queues inside embedded wifi devices we were working on. I was probably not the first to realize that this set of rules probably applies to all kinds of networks, not just ones in embedded device firmware, but I hadn’t really seen it written down before.…

it was without question, one of your more brief, poignant pieces.

Re: Things we finally know about network queues (2017)

#27

Post by dtaht »

kqr wrote: post #2

This is a good collection of recommendations. At a new job, I often end up making changes to internal queues that (it turns out) conform to these recommendations, and get better performance and reliability out of it. Though I didn't learn it by engineering networks like the author -- I took the ideas from lean product development and applied them also to software. > Corollary: limit queue length to a statistically la…

Fair queueing makes a huge difference for a large percentage of traffic, and generally makes an AQM work better by better muxing packets.

I have a long list of things that can be done listed here:

http://www.taht.net/~d/broadcom_aug9_2018.pdf

A new one that has cropped up recently in terms of shortening queues, is rigorous application of the TCP_NOTSENT_LOWAT option, everywhere, but especially in containers.

Re: Things we finally know about network queues (2017)

#28

Post by dtaht »

codeflo wrote: post #12
HALtheWise wrote: post #9

It occurs to me that most of these queue size tradeoffs would be eliminated if the queue operated in a LIFO manner (a stack) instead of FIFO. That way, a burst can naturally get absorbed and re-emitted, but steady state high load doesn't result in increased latency, just packet loss. Can someone with more knowledge of networking enlighten me on why this is a terrible idea?

Let’s say the buffer is nearly full, and the egress rate matches the ingress rate almost exactly. Wouldn’t that mean that the bottom of the stack (the oldest package) never gets transmitted, while the top of the stack is churned constantly? If such a situation persists, the oldest packages might not get transmitted for hours, which means they’d be lost for all practical purposes, while still taking up valuable buffer…

Head drop, so long as it preserves a "round" so a malignant sender cannot force all other traffic out of the queue, is great. This is what fq-codel, fq-pie, and cake do.

Re: Things we finally know about network queues (2017)

#29

Post by rkeene2 »

Bluecobra wrote: post #23
rkeene2 wrote: post #6

I haven't had time to read through the whole text but lots of it rings true with my understanding. I wrote an article that, if you like this one, you will probably also gain from: https://dev.to/rkeene/why-is-there-packet-loss-1p8a

Why wasn’t the speed of the WAN circuit questioned? To paraphrase Google, OC-12 is so slow I forgot how to count that low. If your link is fast enough, you don’t need to worry about QoS, the switch buffer size, or have to buy an expensive WAN traffic shaping device. :)

The link speed of the circuit doesn't indicate the bandwidth of a channel (MPLS or PVC/SVC), and there is no way to know the possible bandwidth since it's a variable (with upper and lower bounds) and so you always have to worry about this (which is why there's packet loss)

Re: Things we finally know about network queues (2017)

#30

Post by zero_iq »

dtaht wrote: post #28
codeflo wrote: post #12

Earlier quoted context omitted.

Let’s say the buffer is nearly full, and the egress rate matches the ingress rate almost exactly. Wouldn’t that mean that the bottom of the stack (the oldest package) never gets transmitted, while the top of the stack is churned constantly? If such a situation persists, the oldest packages might not get transmitted for hours, which means they’d be lost for all practical purposes, while still taking up valuable buffer…

Head drop, so long as it preserves a "round" so a malignant sender cannot force all other traffic out of the queue, is great . This is what fq-codel, fq-pie, and cake do.

Can you clarify what you mean by a "round" here, please? Do you mean a complete round-robin pass of clients (or whatever segmentation method is used) in the fair queue?
Post reply on HN