Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we finally know about network queues (2017)

apenwarr.ca

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Things we finally know about network queues (2017)

#31

Post by toast0 »

HALtheWise wrote: post #9

It occurs to me that most of these queue size tradeoffs would be eliminated if the queue operated in a LIFO manner (a stack) instead of FIFO. That way, a burst can naturally get absorbed and re-emitted, but steady state high load doesn't result in increased latency, just packet loss. Can someone with more knowledge of networking enlighten me on why this is a terrible idea?

This isn't a great idea at the packet level. You get really bad results by intentionally reordering packet in a tcp connection. It's not going to be good for any flow that expects packets to arrive in order for the most part (voip calls etc).

Plus accumulating old packets isn't great. FIFO preserves order, but the typical behavior is to drop incomming packets when the buffer is full (or to make an absurdly sized buffer), when it's usually better to drop older packets than fresh packets.

Re: Things we finally know about network queues (2017)

#32

Post by Bluecobra »

rkeene2 wrote: post #29
Bluecobra wrote: post #23

Earlier quoted context omitted.

Why wasn’t the speed of the WAN circuit questioned? To paraphrase Google, OC-12 is so slow I forgot how to count that low. If your link is fast enough, you don’t need to worry about QoS, the switch buffer size, or have to buy an expensive WAN traffic shaping device. :)

The link speed of the circuit doesn't indicate the bandwidth of a channel (MPLS or PVC/SVC), and there is no way to know the possible bandwidth since it's a variable (with upper and lower bounds) and so you always have to worry about this (which is why there's packet loss)

You realize you can get point to point Ethernet circuits with a fixed speed/fiber path right? I have 100G between my data centers and QoS/buffering is something I don’t need to worry about on my network. Some are these paths are dark fiber and can be easily upgraded to 400G when the time comes.

Re: Things we finally know about network queues (2017)

#33

Post by reidjs »

dtaht wrote: post #24

I tried to leave people laughing, here: https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-... And the online book, freely available and primarily on applying fq_codel to everything (and also sch_cake) is here: https://bufferbloat-and-beyond.net/ In the last decade we've managed to eliminate fifos from most of linux, most 3rd party firmwares notably with openwrt and sqm, ios and OSX. The only major things le…

That was an amazing way of explaining a complex topic. Glad to know there are people much smarter than me keeping the internet alive. Thanks for putting so much work into the demonstration.

Re: Things we finally know about network queues (2017)

#34

Post by Terry_Roll »

moonchild wrote: post #4

Why do we know these things? Why are they true? Does anyone have resources for someone who doesn't know that much about networking justifying these statements?

I found it to be quite educational to get a DDOS carried out on a public ip of mine to see how a firewall device faired. Lets just say, it was quite interesting seeing various networking queues fill up and different daemons trying to cope.

There is certainly an element of fine tuning and knowledge needed across multiple daemons on a firewall device which includes the network stack, not only so that a daemon has the resource to function, but also to play its part in the device function as a whole. When a daemon falls over, it can create a new temporary attack vector, in much the same way as a device rebooting shouldn't be online until everything is loaded and running, but I dont even see that in some switches.

You might find some willing individuals on the dark web who can provide such DDOS services, if you wanted to test some things out.

Re: Things we finally know about network queues (2017)

#35

Post by HALtheWise »

codeflo wrote: post #12
HALtheWise wrote: post #9

It occurs to me that most of these queue size tradeoffs would be eliminated if the queue operated in a LIFO manner (a stack) instead of FIFO. That way, a burst can naturally get absorbed and re-emitted, but steady state high load doesn't result in increased latency, just packet loss. Can someone with more knowledge of networking enlighten me on why this is a terrible idea?

Let’s say the buffer is nearly full, and the egress rate matches the ingress rate almost exactly. Wouldn’t that mean that the bottom of the stack (the oldest package) never gets transmitted, while the top of the stack is churned constantly? If such a situation persists, the oldest packages might not get transmitted for hours, which means they’d be lost for all practical purposes, while still taking up valuable buffer…

Apologies for not making this clear, I was definitely assuming a LIFO buffer would drop the oldest message when full, since dropping the newest message has the issue you describe. In that case, for a prolonged period of ingress slightly over egress, the buffer will contain the N most recently received messages that haven't been transmitted, which seems optimal, but _also_ will have no queueing-induced latency for new messages. If there's a gap in the incoming data stream, a LIFO buffer would emit the most-recent (and most likely to be useful) data first. Most importantly, there's now no performance drawback for making the buffer bigger, so tuning it is way less dependent on your expected operating environment.

One other optimization that would probably make sense in a practical implementation is a "max age" limit on things popped from the stack, so that an old message can't sit around for several seconds if the ingress rate happens to exactly matches egress. This limit can be fairly long though (~1s is fine), since it's just trying to approximate when a higher-level protocol would already have completed a full roundtrip and requested retransmit.

Re: Things we finally know about network queues (2017)

#36

Post by HALtheWise »

toast0 wrote: post #31
HALtheWise wrote: post #9

It occurs to me that most of these queue size tradeoffs would be eliminated if the queue operated in a LIFO manner (a stack) instead of FIFO. That way, a burst can naturally get absorbed and re-emitted, but steady state high load doesn't result in increased latency, just packet loss. Can someone with more knowledge of networking enlighten me on why this is a terrible idea?

This isn't a great idea at the packet level. You get really bad results by intentionally reordering packet in a tcp connection. It's not going to be good for any flow that expects packets to arrive in order for the most part (voip calls etc). Plus accumulating old packets isn't great. FIFO preserves order, but the typical behavior is to drop incomming packets when the buffer is full (or to make an absurdly sized buff…

Can you elaborate more on why applications handle packet reordering badly? Is it just that applications are written to assume in-order packets are the "fast path" and need to use less optimized code when things come in out-of-order, or is there some other heuristics that assume packet reordering is rare that would misfire here?

Re: Things we finally know about network queues (2017)

#37

Post by HALtheWise »

ignoramous wrote: post #13
HALtheWise wrote: post #9

It occurs to me that most of these queue size tradeoffs would be eliminated if the queue operated in a LIFO manner (a stack) instead of FIFO. That way, a burst can naturally get absorbed and re-emitted, but steady state high load doesn't result in increased latency, just packet loss. Can someone with more knowledge of networking enlighten me on why this is a terrible idea?

> ...queue size tradeoffs would be eliminated if the queue operated in a LIFO manner (a stack) instead of FIFO. Kind of. Using 'adaptive lifo' with a variant of CoDel is something Facebook explained they do address tail latency: Most services process queues in FIFO (first-in first-out) order. During periods of high queuing, however, the first-in request has often been sitting around for so long that the user may have…

Switching to LIFO adaptively is a cool idea, thanks for sharing that Facebook is using that at scale, I didn't know that.

Thinking about eliminating bufferbloat was the original line of thinking that made me think of LIFO queues, since the latency properties of a drop-oldest LIFO queue are (I think) only related to the burstiness of the traffic flow, not the volume, and so steady flow cannot ever cause any intermediate node to hold a persistently large queue of messages that will ever be transmitted.

From the perspective of an application transmitting over the network, a steady flow through a saturated link results in packet loss but not latency, while a bursty flow results in small burst-sized latencies to random packets, and some reordering as a result. Because the _downstream_ traffic from one saturated link is typically not very bursty when it hits the next saturated link, negative effects won't accumulate in the same way that latency does in bufferbloat.

Re: Things we finally know about network queues (2017)

#38

Post by HALtheWise »

mdzn wrote: post #10
HALtheWise wrote: post #9

It occurs to me that most of these queue size tradeoffs would be eliminated if the queue operated in a LIFO manner (a stack) instead of FIFO. That way, a burst can naturally get absorbed and re-emitted, but steady state high load doesn't result in increased latency, just packet loss. Can someone with more knowledge of networking enlighten me on why this is a terrible idea?

Wouldn't this cause lots of out-of-order packets?

Yep, but IP doesn't provide an ordering guarantee at the link level anyway, and higher-level protocols have sequence numbers so they can reassemble things correctly. If those higher-level mechanisms aren't up for the task of handling lots of small reorderings, that would obviously be an issue.

Re: Things we finally know about network queues (2017)

#40

Post by rkeene2 »

Bluecobra wrote: post #32
rkeene2 wrote: post #29

Earlier quoted context omitted.

The link speed of the circuit doesn't indicate the bandwidth of a channel (MPLS or PVC/SVC), and there is no way to know the possible bandwidth since it's a variable (with upper and lower bounds) and so you always have to worry about this (which is why there's packet loss)

You realize you can get point to point Ethernet circuits with a fixed speed/fiber path right? I have 100G between my data centers and QoS/buffering is something I don’t need to worry about on my network. Some are these paths are dark fiber and can be easily upgraded to 400G when the time comes.

I do, of course, know about the possibility of changing the characteristics of a channel -- it's very likely that this question is not meant to be inflammatory but because it's such a basic thing that I cannot read it any other way.

For what it's worth, at the time you could not get 100 Gbps links, but it's unrelated to the fundamental queuing problem and how they are solved in packet switched networks (by dropping packets in the best-effort systems such as IP/Ethernet and by managing credits/counters in guaranteed bandwidth systems like Fibre Channel/ATM).

You will still have packet loss if you try to send packets at 600 Gbps in the case of a channel whose upper and lower bounds of capacity are 400 Gbps. There are no infinity capacity channels.

Since you have likely more than 400 systems connected at 1Gbps in each data center, you have over provisioned that 400 Gbps link and if each node does 1Gbps you will have packet loss. The value of each packet is probably not equal and so it may make sense to do some kind of QoS for those scenarios (or not, I certainly don't have enough information to answer). This is a problem you can have at any time. If you are in data center operations, you'll do capacity planning to try to mitigate this, but it's still a problem (until there are no overprovisioned paths, which is wasteful and bad engineering in most circumstances).

The point of the essay wasn't to describe the scale of a data center, it was to talk about packet loss in a network with queues using a system that was designed around this, and what that means for users of the network.

Post reply on HN