Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Kernel vs. DPDK: HTTP Performance Showdown

talawah.io

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#51

Post by grive »

dclusin wrote: post #26
jaimex2 wrote: post #21

Earlier quoted context omitted.

Basically its what people who want to bypass the kernel network stack because they think its slow. They then spend the next few years writing their own stack till they realise they've just re-written what the kernel does and its slower and full of exploits. Yeah, receiving packets is fast when you aren't doing anything with them.

It’s typically done when end to end latency is more important than full protocol compliance and hardening against myriad types of attackers. High frequency trading is typically where the applications where you see these sorts of implementations being really compelling. For these types of applications there are proprietary implementations that you can buy from vendors that are more suited to latency sensitive applicat…

FPGAs are falling out of favor, ASICs are sufficient to implement most network processing.

Funnily, one of the biggest DPDK feature is an API to program smartNICs exactly in that way.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#52

Post by gjulianm »

jaimex2 wrote: post #21
0xbadcafebee wrote: post #9

For those like me going "......what is dpdk" The Data Plane Development Kit (DPDK) is an open source software project managed by the Linux Foundation. It provides a set of data plane libraries and network interface controller polling-mode drivers for offloading TCP packet processing from the operating system kernel to processes running in user space. This offloading achieves higher computing efficiency and higher pac…

Basically its what people who want to bypass the kernel network stack because they think its slow. They then spend the next few years writing their own stack till they realise they've just re-written what the kernel does and its slower and full of exploits. Yeah, receiving packets is fast when you aren't doing anything with them.

That's not the reality. The kernel network stack does a lot of things that for some applications are not needed, so DPDK is used for those cases when they need very high performance. For example, packet capture, routing, packet generation... I don't think I've seen anyone rewriting what the kernel does precisely because when you use DPDK you don't want to do what the kernel does.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#53

Post by grive »

jaimex2 wrote: post #21
0xbadcafebee wrote: post #9

For those like me going "......what is dpdk" The Data Plane Development Kit (DPDK) is an open source software project managed by the Linux Foundation. It provides a set of data plane libraries and network interface controller polling-mode drivers for offloading TCP packet processing from the operating system kernel to processes running in user space. This offloading achieves higher computing efficiency and higher pac…

Basically its what people who want to bypass the kernel network stack because they think its slow. They then spend the next few years writing their own stack till they realise they've just re-written what the kernel does and its slower and full of exploits. Yeah, receiving packets is fast when you aren't doing anything with them.

Networking stacks are used in two ways: endpoints and forwarders.

Forwarders are usually not doing much with packets, just reading a few fields and choosing an output port. These are very common function in network cores (telcos, datacenters, ...).

DPDK is not well-suited for endpoint application programming, even though here you can still squeeze some additional performance.

But don't dismiss a framework that is currently so widely deployed just because you are not familiar with its most common use-case.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#54

Post by gjulianm »

pclmulqdq wrote: post #3

This was a fascinating read and the kernel does quite nicely in comparison - 66% of DPDK performance is amazing. That said, the article completely nails the performance advantage: DPDK doesn't do a lot of stuff that the kernel does. That stuff takes time. If I recall correctly, DPDK abstractions themselves cost a bit of NIC performance, so it might be interesting to see a comparison including a raw NIC-specific kerne…

> If I recall correctly, DPDK abstractions themselves cost a bit of NIC performance, so it might be interesting to see a comparison including a raw NIC-specific kernel bypass framework (like the SolarFlare one).

DPDK performs fairly well, even better for the most part. For some years I maintained a modification of the ixgbe kernel driver for Intel NICs that allowed us to perform high-performance traffic capture. We finally moved to DPDK once it was stable enough and we had the need to support more NICs, and in our comparisons we didn't see a performance hit.

Maybe manufacturer-made drivers can be better than DPDK, but if I had to guess that would be not because of the abstractions but because of the knowledge of the NIC architecture and parameters. I remember when we tried to do a PoC of a Mellanox driver modification and a lot of the work to get high performance was understanding the NIC options and tweaking them to get the most for our use case.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#55

Post by baruch »

touisteur wrote: post #43
monocasa wrote: post #30

Earlier quoted context omitted.

eBPF pushes computation into the kernel to be colocated with kernel data/events. DPDK pushes all of the relevant data (the network packets) into user space to be colocated with user data/events. You can mix both a bit to make the selection of where packets go with a eBPF program, and running higher level stacks in user space where user space is still dealing with raw packets rather than sockets.

And now you're in AF_XDP territory, which is an alternative to dpdk, a tentative to normalise kernel 'bypass' for network operations.

DPDK has an XDP driver too so you can use that and also work with network cards that do not support XDP or the zero copy variant for more flexibility. Though DPDK can be a pain in and of itself, it is fairly opinionated and I don't like all of its opinions. It has its roots in switching and routing applications and using it for an endpoint can have some mismatches in opinion. It is open source so you can modify it which works out for me in the end.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#56

Post by baruch »

jaimex2 wrote: post #21
0xbadcafebee wrote: post #9

For those like me going "......what is dpdk" The Data Plane Development Kit (DPDK) is an open source software project managed by the Linux Foundation. It provides a set of data plane libraries and network interface controller polling-mode drivers for offloading TCP packet processing from the operating system kernel to processes running in user space. This offloading achieves higher computing efficiency and higher pac…

Basically its what people who want to bypass the kernel network stack because they think its slow. They then spend the next few years writing their own stack till they realise they've just re-written what the kernel does and its slower and full of exploits. Yeah, receiving packets is fast when you aren't doing anything with them.

Normally the endpoint stack you need is more limited initially and you don't have to have all the baggage. The biggest advantage you get at the end is zero copy network and if what you do involves tons of reading and writing to the network you can definitely gain performance in both bandwidth and latency by better using your CPU.

You can do XDP as well and gain zero copy but for a small subset of NICs and you still need to implement you own network stack I don't see ac way to avoid it much. There are existing TCP stacks for DPDK that you can use as well.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#57

Post by evgpbfhnr »

anonymoushn wrote: post #36
evgpbfhnr wrote: post #5

At the point you've gotten syscall overhead is definitely going to be a big thing (even without spectre mitigations enabled) -- I'd be very curious to see how far a similar io_uring benchmark would get you. It supports IOPOLL (polling of the socket) and SQPOLL (kernel side polling of the request queue) so hopefully the fact that application driving it is in another thread wouldn't slow it too much... With multi-shot…

IOPOLL is for disks. I think that with very recent kernels you will get busy polling of the socket with just SQPOLL. See here: https://github.com/axboe/liburing/issues/345#issuecomment-10...

oh! That's not obvious at all from the man page (io_uring_enter.2)

If the io_uring instance was configured for polling, by specifying IORING_SETUP_IOPOLL in the call to io_uring_setup(2), then min_complete has a slightly different meaning. Passing a value of 0 instructs the kernel to return any events which are already complete, without blocking. If min_complete is a non-zero value, the kernel will still return immediately if any completion events are available. If no event completions are available, then the call will poll either until one or more completions become available, or until the process has ex‐ ceeded its scheduler time slice.

... Well, TIL -- thanks! and the NAPI patch you pointed at looks interesting too.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#58

Post by gjulianm »

touisteur wrote: post #43
monocasa wrote: post #30

Earlier quoted context omitted.

eBPF pushes computation into the kernel to be colocated with kernel data/events. DPDK pushes all of the relevant data (the network packets) into user space to be colocated with user data/events. You can mix both a bit to make the selection of where packets go with a eBPF program, and running higher level stacks in user space where user space is still dealing with raw packets rather than sockets.

And now you're in AF_XDP territory, which is an alternative to dpdk, a tentative to normalise kernel 'bypass' for network operations.

I wouldn't say XDP is an alternative to DPDK. XDP severely limits what you can do as you insert code inside of the kernel. DPDK exposes the NIC to regular user code so you can do what you want. XDP is useful when you want to modify what the kernel does with packets before they go to their destination application, DPDK when you want to do whatever with the packets without sending them to any other application.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#59

Post by jackbondpreston »

bitcharmer wrote: post #40
medawsonjr wrote: post #18

Earlier quoted context omitted.

Or better yet, Mellanox VMA since it's open source (unlike Solarflare OpenOnload) and the NICs are far less expensive.

That's not true. OpenOnload is open source: https://github.com/majek/openonload

https://github.com/xilinx-cns/onload

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#60

Post by gpderetta »

jackbondpreston wrote: post #59
bitcharmer wrote: post #40

Earlier quoted context omitted.

That's not true. OpenOnload is open source: https://github.com/majek/openonload

https://github.com/xilinx-cns/onload

It as been OSS since forever, but I thought that there were some patent gotchas. In any case it now apparently even support non-SolarFlare NICs!

Onload has also has other nice features like accelerating machine-local stuff (pipes, unix sockets and other stuff).

Post reply on HN