Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The CPU Cost of Networking on a Linux Host

people.kernel.org

Post reply on HN
Print view View original

21–30 of 50 posts

Re: The CPU Cost of Networking on a Linux Host

#21

Post by dsr_ »

Aaronstotle wrote: post #18
dahfizz wrote: post #3

This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…

I'm curious, what makes the natting/routing performance much better on a router? Is it that the router is an ASIC designed only to process packets vs a cpu that can execute any code?

In particular, a modern high-end router has a CPU (which might be an x86 of some kind) which handles administrative and auxiliary tasks (NTP, ssh, DHCP reflection, whatever you want) while making one or more dedicated ASICs handle packets. When you manage the router, you're actually talking to the supervisory CPU.

Re: The CPU Cost of Networking on a Linux Host

#22

Post by Matthias247 »

dahfizz wrote: post #10
onei wrote: post #8

Earlier quoted context omitted.

How did you get to six? Is that just sending a single packet without dealing with responses? Never thought about this before so I'm curious.

Source container -> VM -> host -> host -> VM -> destination container

Correct me if I'm wrong, but I don't think the container transition introduces any additional copies and hops. Containers are not VMs. They just use isolation that the kernel enforces. Whether you send a packet from inside or outside a container to the network should be exactly the same amount of work unless you opt-in to run some extra software-level NAT step.

Re: The CPU Cost of Networking on a Linux Host

#23

Post by toast0 »

sleepydog wrote: post #14
dahfizz wrote: post #10

Earlier quoted context omitted.

Source container -> VM -> host -> host -> VM -> destination container

It's not as bad as it sounds as a few of those transitions can be implemented via zero-copy APIs, so you're really just passing around pointers to shared memory.

If there's NAT involved (as was posited in the start of the thread), then you can't just copy, you also need to modify.

Re: The CPU Cost of Networking on a Linux Host

#24

Post by oneplane »

dahfizz wrote: post #16
dan_quixote wrote: post #5

Earlier quoted context omitted.

> That's dumb. Is it? It's certainly inefficient compared to dedicated hardware. But so is anything relying on a CPU - we could just use ASICs for everything. But then every logical change requires weeks/months/years of development and manufacturing. The goal of k8s, VMs, etc is flexibility. I can set up a 100-node k8s cluster with less-than-perfectly-efficient networking stack in mere minutes. Good luck matching tha…

> But so is anything relying on a CPU - we could just use ASICs for everything. But then every logical change requires weeks/months/years of development and manufacturing. Right, except the asics already exist and you're actively choosing the less efficient, more expensive option. > The goal of k8s, VMs, etc is flexibility. I don't think this is necessarily bad, as long as you understand the tradeoff. You're choosing…

In this case it's not even user experience vs. developer experience as you can have both; it's just the cost for performance gets higher as the efficiency decreases.

On the other hand: the cost for development goes down if you don't need to pay for extra steps taken by an extra person. If you take a 10-step process that is run by 5 people and reduce it to a 5-step process run by 3, you have 2 more people to do other stuff, or roles that you don't have to create/fill in the first place.

Re: The CPU Cost of Networking on a Linux Host

#25

Post by EvanAnderson »

ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

I wonder if there's an economic argument, at useful scales, to using FPGA's in general purpose servers to accelerate network performance. The purpose-built ASICs would win on cost-per-unit every time, but the FPGA would have some adaptability to new protocols or algorithms that the ASIC wouldn't.

Re: The CPU Cost of Networking on a Linux Host

#26

Post by tw04 »

ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

That's not entirely accurate - Cisco open sourced VPP which they use in many products:

https://blogs.cisco.com/sp/a-bigger-helping-of-internet-plea...

https://docs.fd.io/vpp/16.09/

While they still obviously make custom ASICs, they've moved towards more software where possible.

Re: The CPU Cost of Networking on a Linux Host

#27

Post by wmf »

EvanAnderson wrote: post #25
ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

I wonder if there's an economic argument, at useful scales, to using FPGA's in general purpose servers to accelerate network performance. The purpose-built ASICs would win on cost-per-unit every time, but the FPGA would have some adaptability to new protocols or algorithms that the ASIC wouldn't.

Yes, Azure uses FPGAs for networking. Programmable NPUs like Netronome or Pensando are a middle ground that are more cost-efficient than FPGAs for most needs.

Re: The CPU Cost of Networking on a Linux Host

#28

Post by shanemhansen »

Matthias247 wrote: post #22
dahfizz wrote: post #10

Earlier quoted context omitted.

Source container -> VM -> host -> host -> VM -> destination container

Correct me if I'm wrong, but I don't think the container transition introduces any additional copies and hops. Containers are not VMs. They just use isolation that the kernel enforces. Whether you send a packet from inside or outside a container to the network should be exactly the same amount of work unless you opt-in to run some extra software-level NAT step.

You're wrong and right, because "container" is ill-defined. If you run your container in the host network namespace, then there's zero overhead. It's more common to have your container's "nic" be one end of a veth pair. That's not free.

Finally many platforms like docker, in the past, will spin up userspace proxies (docker-proxy) which is even more expensive.

Re: The CPU Cost of Networking on a Linux Host

#29

Post by ra1n85 »

tw04 wrote: post #26
ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

That's not entirely accurate - Cisco open sourced VPP which they use in many products: https://blogs.cisco.com/sp/a-bigger-helping-of-internet-plea... https://docs.fd.io/vpp/16.09/ While they still obviously make custom ASICs, they've moved towards more software where possible.

Sorry, what's not accurate here?

Re: The CPU Cost of Networking on a Linux Host

#30

Post by Hikikomori »

tw04 wrote: post #26
ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

That's not entirely accurate - Cisco open sourced VPP which they use in many products: https://blogs.cisco.com/sp/a-bigger-helping-of-internet-plea... https://docs.fd.io/vpp/16.09/ While they still obviously make custom ASICs, they've moved towards more software where possible.

Pretty much everything above their small medium business ISR routers use ASIC/NPU for all traffic possible. Which routers are you referring to?
Post reply on HN