Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The CPU Cost of Networking on a Linux Host

people.kernel.org

Post reply on HN
Print view View original

31–40 of 50 posts

Re: The CPU Cost of Networking on a Linux Host

#31

Post by Klinky »

dahfizz wrote: post #3

This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…

Do most scenarios require that pedal-to-the-metal level of performance? A lot of applications are CPU-bound either within the application server or a dependent service(database). Your payloads/throughput may actually be rather low, and the inefficiency may be a tiny percentage of CPU usage compared to the application itself.

If you need nonstop CDN levels of bandwidth, then you're probably going to go with a setup that has fewer layers, but CRUD applications don't exactly need terabits of bandwidth.

Re: The CPU Cost of Networking on a Linux Host

#32

Post by bogomipz »

ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

>"In the last 15 years there has been a hard move away from these architectures."

It's been longer than that for router vendors. The move to hardware based forwarding was in the 90s.

Re: The CPU Cost of Networking on a Linux Host

#33

Post by yummypaint »

EvanAnderson wrote: post #25
ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

I wonder if there's an economic argument, at useful scales, to using FPGA's in general purpose servers to accelerate network performance. The purpose-built ASICs would win on cost-per-unit every time, but the FPGA would have some adaptability to new protocols or algorithms that the ASIC wouldn't.

Ive been waiting for mainstream workstation motherboards with this capability for years. Presumably a pci card is how this would be handled in practice for now. My naive take is that toolchains are still too convoluted and bogged down with licensing schemes to deliver the kind of real-time highly integrated adaptability this would require for individual users. Would be great if the barrier to entry has in fact dropped enough that university-sized networks could consider them.

Re: The CPU Cost of Networking on a Linux Host

#34

Post by karatekidd32v »

EvanAnderson wrote: post #25
ra1n85 wrote: post #2

For many years, the most popular routing platforms (i.e., boxes built by Cisco) performed IP packet forwarding and management functions on the same processor (often a RISC architecture). In cases where packet rate was high, it was possible for devices to become unresponsive or lose critical protocols responsible for sharing routing information. In the last 15 years there has been a hard move away from these architect…

I wonder if there's an economic argument, at useful scales, to using FPGA's in general purpose servers to accelerate network performance. The purpose-built ASICs would win on cost-per-unit every time, but the FPGA would have some adaptability to new protocols or algorithms that the ASIC wouldn't.

There's some cool tech going on at barefoot (https://barefootnetworks.com/)

I don't know much about the technical details, but the pitch I've heard is that it gives you ASIC level performance with more flexibility to reprogram the chip (not full FPGA).

Re: The CPU Cost of Networking on a Linux Host

#35

Post by dahfizz »

Klinky wrote: post #31
dahfizz wrote: post #3

This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…

Do most scenarios require that pedal-to-the-metal level of performance? A lot of applications are CPU-bound either within the application server or a dependent service(database). Your payloads/throughput may actually be rather low, and the inefficiency may be a tiny percentage of CPU usage compared to the application itself. If you need nonstop CDN levels of bandwidth, then you're probably going to go with a setup th…

As the article explains, it's not all about throughput or latency (though those do suffer), it's also about how much CPU time you spend because of the network architecture. If you're running a cpu bound app, all the more reason to avoid sending 6+ interupts per packet.

Re: The CPU Cost of Networking on a Linux Host

#36

Post by rayiner »

Aaronstotle wrote: post #18
dahfizz wrote: post #3

This is one of the reasons I have a strong aversion to "cloud" technologies like docker and kubernetes. You take networking, something with decades of development and hardware support, and you put it all in the CPU. To be clear, Linux has a very robust networking stack. But it will never come close to the natting and routing performance of an actual router. And so we develop things like DPDK to spend even more CPU ju…

I'm curious, what makes the natting/routing performance much better on a router? Is it that the router is an ASIC designed only to process packets vs a cpu that can execute any code?

That's not really the whole explanation. There are a couple of things in play. First, while Linux can perform routing, it's mainly intended to be an end host. So for example software like VPP that is designed with the assumption that it can use polling and burn up entire CPU cores to optimize packet forwarding can perform 10x faster on the same hardware.

Second, router ASICs can be designed to sped transistors on a predictable packet flow path, instead of intelligence like out-of-order execution to make general purpose code fast. For example, routers have big expensive content-addressable memories called TCAMs that are used to store things like routing tables and ACLs. The ASIC, moreover, can implement a highly tuned pipeline designed around the latencies in the underlying memories. E.g. you get a packet, grab the destination address, look up the next hop in the routing table, etc. Each step takes a predictable amount of time that you can account for and optimize.

Third, parallelism is much cheaper in hardware than in general purpose CPUs. It takes a lot more transistors to be able to execute a second general-purpose instruction stream than to have a single-purpose circuit that does some work in parallel with something else.

Re: The CPU Cost of Networking on a Linux Host

#37

Post by dahfizz »

Matthias247 wrote: post #22
dahfizz wrote: post #10

Earlier quoted context omitted.

Source container -> VM -> host -> host -> VM -> destination container

Correct me if I'm wrong, but I don't think the container transition introduces any additional copies and hops. Containers are not VMs. They just use isolation that the kernel enforces. Whether you send a packet from inside or outside a container to the network should be exactly the same amount of work unless you opt-in to run some extra software-level NAT step.

You are correct that containers can be implemented that way, but docker containers aren't. They use NATing / bridging / veth pairs, which is much more usable but also much slower.

Re: The CPU Cost of Networking on a Linux Host

#38

Post by otterley »

dahfizz wrote: post #10
onei wrote: post #8

Earlier quoted context omitted.

How did you get to six? Is that just sending a single packet without dealing with responses? Never thought about this before so I'm curious.

Source container -> VM -> host -> host -> VM -> destination container

This hasn't been the case for years. See, e.g., SR-IOV, which allows modern PCIe network adapters to completely bypass a host virtual switch and present itself directly to a VM.

https://www.intel.com/content/dam/doc/application-note/pci-s... is a good overview.

Also, at least on AWS, you can directly attach VPC network interfaces to containers, which (among other things) obviates the need for software bridging, veth pairs, etc.

Re: The CPU Cost of Networking on a Linux Host

#39

Post by ra1n85 »

karatekidd32v wrote: post #34
EvanAnderson wrote: post #25

Earlier quoted context omitted.

I wonder if there's an economic argument, at useful scales, to using FPGA's in general purpose servers to accelerate network performance. The purpose-built ASICs would win on cost-per-unit every time, but the FPGA would have some adaptability to new protocols or algorithms that the ASIC wouldn't.

There's some cool tech going on at barefoot ( https://barefootnetworks.com/ ) I don't know much about the technical details, but the pitch I've heard is that it gives you ASIC level performance with more flexibility to reprogram the chip (not full FPGA).

Yeah, it's an interesting approach. They're basically allowing you to define packet processing with P4 on their Tofino family of chipsets: https://p4.org/

That said, there's only so much you can do in a chip before considerable tradeoffs are going to be made. They're not going to offer the same level of flexibility you get out of a general purpose CPU, but may not have same the restrictions of most fixed pipeline chips - their product sits somewhere in the middle. Also, P4 seems to sit in a space complex enough to make it unreasonable for most network shops - it's not for your average enterprise or service provider network.

Post reply on HN