The CPU Cost of Networking on a Linux Host
people.kernel.org
The CPU Cost of Networking on a Linux Host
1–10 of 50 posts
Re: The CPU Cost of Networking on a Linux Host
#2For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information.
In the last 15 years there has been a hard move away from these architectures. Almost no packets are forwarded by the same processors running management and control plane functions anymore. This is mainly because the required traffic rates today need dedicated silicon purpose built for the task (the Broadcom Tomahawk3 can do 12.8 Terabits/sec above a relatively small packet size).
I don't know how things will shake out for the Linux world and x86 packet forwarding given the trend and lack of real performance in the kernel. Right now, your best bet when it comes to Linux and high network throughput/packet processing requirements is to just bypass the kernel entirely with DPDK, a "smart" NIC, or XDP.
Re: The CPU Cost of Networking on a Linux Host
#3This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU.
To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router.
And so we develop things like DPDK to spend even more CPU just to keep things usable, but it still feels like a big step backwards.
A typical k8s deployment runs in containers that are in VMs. So each packet you want to send to or from a container needs to touch a cpu and traverse a networking stack six times. That's dumb.
Re: The CPU Cost of Networking on a Linux Host
#4This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…
Interesting. I always suspected something like this was going on. Virtual networking / "SDN" just sounded slow to me.
Do you know a good source, to learn about the lower-level Linux/container/docker/k8s networking?
Re: The CPU Cost of Networking on a Linux Host
#5This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…
> That's dumb.
Is it? It's certainly inefficient compared to dedicated hardware. But so is anything relying on a CPU - we could just use ASICs for everything. But then every logical change requires weeks/months/years of development and manufacturing.
The goal of k8s, VMs, etc is flexibility. I can set up a 100-node k8s cluster with less-than-perfectly-efficient networking stack in mere minutes. Good luck matching that with dedicated hardware.
Re: The CPU Cost of Networking on a Linux Host
#6This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…
Interesting. I always suspected something like this was going on. Virtual networking / "SDN" just sounded slow to me. Do you know a good source, to learn about the lower-level Linux/container/docker/k8s networking?
I got the most value poking around with network namespaces manually via the "ip" command. There are also true VRFs in Linux these days which are much lighter but not nearly as common, most everything is a namespace with either bridging or NAT and some fancy configuration interface to drive that state.
Re: The CPU Cost of Networking on a Linux Host
#7This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…
This is partly true and partly not. All the physical routing and switching is still there and it's more efficient than ever. What we've done is added more functionality. Additional layers of abstraction like VMs or containers are generally going to have some performance cost.
It's possible to optimize basic packet forwarding in a hypervisor/containervisor using leaner options like ipvlan or even SR-IOV. If you replace hardware firewalls with something like k8s network policy then you are technically replacing hardware with software but it isn't that slow if configured right (hello Cilium) and you're probably implementing a more sophisticated policy anyway.
Re: The CPU Cost of Networking on a Linux Host
#8This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…
How did you get to six? Is that just sending a single packet without dealing with responses? Never thought about this before so I'm curious.
Re: The CPU Cost of Networking on a Linux Host
#9This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…
Given clouds are like mainframes reinvented, you might like reading on how the mainframes handled I/O:
https://en.wikipedia.org/wiki/Channel_I/O
I always thought PC's could be setup to do something similar. The cores used for that could even be small and simple compared to main cores. Folks in early 2001 could be Bittorrenting their Linux distros or whatever they use it for with no user-visible lag. I'd like at least one each for user-input devices, graphics, storage, and networking. Other good reasons to isolate them a bit from each other.
There's already ARM SoC's in embedded that have a good core for apps and weak one for I/O. Recently, the RISC-V chip with the minion cores. If embedded can do it, then there seems to be no technical limitation holding back desktops. Just marketing, backward compatibility, etc.
Re: The CPU Cost of Networking on a Linux Host
#10This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…
How did you get to six? Is that just sending a single packet without dealing with responses? Never thought about this before so I'm curious.
Source container -> VM -> host -> host -> VM -> destination container