Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The CPU Cost of Networking on a Linux Host

people.kernel.org

Post reply on HN
Print view View original

41–50 of 50 posts

Re: The CPU Cost of Networking on a Linux Host

#41

Post by cheeseprocedure »

heinrichhartman wrote: post #4
dahfizz wrote: post #3

This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…

Interesting. I always suspected something like this was going on. Virtual networking / "SDN" just sounded slow to me. Do you know a good source, to learn about the lower-level Linux/container/docker/k8s networking?

packagecloud's in-depth writeups on the Linux networking stack are a great read:

https://blog.packagecloud.io/eng/2016/06/22/monitoring-tunin...

https://blog.packagecloud.io/eng/2017/02/06/monitoring-tunin...

The illustrated guide to receiving data is also solid:

https://blog.packagecloud.io/eng/2016/10/11/monitoring-tunin...

Re: The CPU Cost of Networking on a Linux Host

#42

Post by jes »

ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

I was intrigued by your comment about the Tomahawk3. Turns out there is a Tomahawk4 implemented in 7nm which has twice the performance. Thank you for tipping me off to these devices:

https://www.broadcom.com/company/news/product-releases/52756

Re: The CPU Cost of Networking on a Linux Host

#43

Post by neurostimulant »

dahfizz wrote: post #3

This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…

I think I read somewhere a while ago that kubernetes cluster will start experiencing big slowdown at around 500 pods due to overhead of its internal networking components. Not sure if it's still true now. I'm going to deploy a cluster that might contains more than 500 pods soon so I'm not really eager to find out the hard way.

Re: The CPU Cost of Networking on a Linux Host

#44

Post by ra1n85 »

jes wrote: post #42
ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

I was intrigued by your comment about the Tomahawk3. Turns out there is a Tomahawk4 implemented in 7nm which has twice the performance. Thank you for tipping me off to these devices: https://www.broadcom.com/company/news/product-releases/52756

Yep, though it’s not commercially available quite yet. Likely only shipping samples to switch manufacturers at this time.

Re: The CPU Cost of Networking on a Linux Host

#45

Post by rstuart4133 »

ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

> I don't know how things will shake out for the Linux world and x86 packet forwarding given the trend and lack of real performance in the kernel.

That's pretty much settled isn't it? It was settled in the same way the "it got too much for the general purpose CPU" problem always gets settled - you do it in custom silicon, and define a standardised interface.

That's pretty much what happened with 3D graphics where the standardised interface was OpenGL (but now seems to be up in the air). It's also pretty much what's happened in AI with standardised libraries interfacing to custom hardware, and it's what happened with networking.

For networking, the interface standard is OpenFlow. So, if you think it's possible you will need to handle links of about 1Gbs or over in the future, you do your networking using an OpenFlow implementation like Faucet. If it's not much above 100Mb/s the Linux kernel module that implements OpenFlow, called openvswitch, will be fine. Otherwise use some custom hardware.

Openvswitch has been around since 2009 - so it's not exactly a new thing.

Re: The CPU Cost of Networking on a Linux Host

#46

Post by gnufx »

ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

Are those kernel bypasses better than the Linux-based RDMA we've had for ages (on appropriate fabrics)?

Re: The CPU Cost of Networking on a Linux Host

#47

Post by gnufx »

dan_quixote wrote: post #5
dahfizz wrote: post #3

This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…

> That's dumb. Is it? It's certainly inefficient compared to dedicated hardware. But so is anything relying on a CPU - we could just use ASICs for everything. But then every logical change requires weeks/months/years of development and manufacturing. The goal of k8s, VMs, etc is flexibility. I can set up a 100-node k8s cluster with less-than-perfectly-efficient networking stack in mere minutes. Good luck matching tha…

Experimentally, you don't need luck to set up ~200 dedicated hardware nodes in minutes in an HPC cluster, for instance. (They do have to be connected up, with known or discoverable MAC addresses.)

Re: The CPU Cost of Networking on a Linux Host

#48

Post by CountSessine »

dahfizz wrote: post #16
dan_quixote wrote: post #5

Earlier quoted context omitted.

> That's dumb. Is it? It's certainly inefficient compared to dedicated hardware. But so is anything relying on a CPU - we could just use ASICs for everything. But then every logical change requires weeks/months/years of development and manufacturing. The goal of k8s, VMs, etc is flexibility. I can set up a 100-node k8s cluster with less-than-perfectly-efficient networking stack in mere minutes. Good luck matching tha…

> But so is anything relying on a CPU - we could just use ASICs for everything. But then every logical change requires weeks/months/years of development and manufacturing. Right, except the asics already exist and you're actively choosing the less efficient, more expensive option. > The goal of k8s, VMs, etc is flexibility. I don't think this is necessarily bad, as long as you understand the tradeoff. You're choosing…

Right, except the asics already exist and you're actively choosing the less efficient, more expensive option.

It’s hard to imagine any dev task being more expensive than organizing and programming tables into a trident or tomahawk chip using the broadcom’s SDK.

Re: The CPU Cost of Networking on a Linux Host

#49

Post by wmf »

gnufx wrote: post #46
ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

Are those kernel bypasses better than the Linux-based RDMA we've had for ages (on appropriate fabrics)?

One problem with RDMA is that it's not compatible with existing protocols.

Re: The CPU Cost of Networking on a Linux Host

#50

Post by gnufx »

wmf wrote: post #49
gnufx wrote: post #46

Earlier quoted context omitted.

Are those kernel bypasses better than the Linux-based RDMA we've had for ages (on appropriate fabrics)?

One problem with RDMA is that it's not compatible with existing protocols.

It's existed for a fair while as Infiniband doing Linux by-pass; the comment didn't say TCP/IP. (The general technique pre-dates Linux, of course.)
Post reply on HN