Linux Kernel vs. DPDK: HTTP Performance Showdown • Hacker News Board

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#41

Post by thekozmo » Tue, Jul 05, 2022, 6:11 AM UTC

BeefWellington wrote: post #29

talawahtech wrote: post #24

Earlier quoted context omitted.

Yea, it is definitely a fake HTTP server which I acknowledge in the article [1]. However based on the size of the requests, and my observation of the number of packets per second in/out being symmetrical at the network interface level, I didn't have a concern about doubled responses. Skipping the parsing of the HTTP requests definitely gives a performance boost, but for this comparison both sides got the same boost,…

> Yea, it is definitely a fake HTTP server which I acknowledge in the article It's not actually an HTTP server though... For these purposes, it's essentially no more useful than netcat dumping out a preconfigured text file. Titling it "HTTP Performance showdown" is doubly bad here since there's no real-world (or even moderately synthetic) HTTP requests happening; you just always get the same static set of data for ev…

ScyllaDB uses Seastar as an engine and the DynamoDB compatible API use HTTP parsing, so this use case is real. Of course the DB has much more to do than this benchmark with a static http reply but Scylla also uses many more core in the server, thus it is close to real life. We do use the kernel's tcp stack, due to all of its features and also since we don't have capacity for a deeper analysis.

Some K/V workloads are affected by the networking stack and we recently seen issues if we chose not the ideal interrupt mode (multiqueue vs single queue in small machines)

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#42

Post by thekozmo » Tue, Jul 05, 2022, 6:16 AM UTC

What's amazing is that the seastar tcp stack hasn't been changed over the past 7 years, while the kernel received plenty of improvements (in order to close the gap vs kernel bypass mechanisms). Still, for >> 99% of users, there is no need to bypass the kernel.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#43

Post by touisteur » Tue, Jul 05, 2022, 6:20 AM UTC

monocasa wrote: post #30

harpratap wrote: post #23

Earlier quoted context omitted.

How does this differ from eBPF?

eBPF pushes computation into the kernel to be colocated with kernel data/events. DPDK pushes all of the relevant data (the network packets) into user space to be colocated with user data/events. You can mix both a bit to make the selection of where packets go with a eBPF program, and running higher level stacks in user space where user space is still dealing with raw packets rather than sockets.

And now you're in AF_XDP territory, which is an alternative to dpdk, a tentative to normalise kernel 'bypass' for network operations.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#44

Post by touisteur » Tue, Jul 05, 2022, 6:36 AM UTC

jaimex2 wrote: post #21

0xbadcafebee wrote: post #9

For those like me going "......what is dpdk" The Data Plane Development Kit (DPDK) is an open source software project managed by the Linux Foundation. It provides a set of data plane libraries and network interface controller polling-mode drivers for offloading TCP packet processing from the operating system kernel to processes running in user space. This offloading achieves higher computing efficiency and higher pac…

Basically its what people who want to bypass the kernel network stack because they think its slow. They then spend the next few years writing their own stack till they realise they've just re-written what the kernel does and its slower and full of exploits. Yeah, receiving packets is fast when you aren't doing anything with them.

Sometimes you really don't do much more on packets than receive and store in (e.g) ring buffers until you can trigger some computation on aggregated data. Sometimes your application has critical latency and you need very light parsing. Not everyone uses tcp and all the options, some people are just down to jumbo udp packets, no fragmentation, and 800Gb/s rx...

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#45

Post by touisteur » Tue, Jul 05, 2022, 6:50 AM UTC

I feel this would be a good place to use a spark-based TCP stack. You're bypassing the kernel, have to run stuff as root or risky CAP_ rights, your stack should be as solid as possible.

https://www.adacore.com/papers/layered-formal-verification-o...

Might also give people here some ideas on how to combine symbolic execution, proof, C and SPARK code and how to gain confidence in each part of a network stack.

I think there's even some ongoing work climbing up the stack up to HTTP but not sure of the plan (not involved).

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#46

Post by thekozmo » Tue, Jul 05, 2022, 7:03 AM UTC

limoce wrote: post #15

I am not 100% sure that all of the mitigation overhead comes from syscalls, but it stands to reason that a lot of it arises from security hardening in user-to-kernel and kernel-to-user transitions. Will io_uring be also affected by Spectre mitigations given it has eliminated most kernel/user switches? And did anyone do a head-to-head comparison between io_uring and DPDK?

Good point. This is more of a tcp stack comparison between the kernel and userspace. Seastar has a sharded (per core) stack, which is very beneficial when the number of threads is high

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#47

Post by fefe23 » Tue, Jul 05, 2022, 7:31 AM UTC

Why is this interesting to anyone? Haven't we all moved to https by now?

Optimizing raw http seems to me like a huge waste of time by now. I say that as someone who has spent years optimizing raw http performance. None of that matters these days.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#48

Post by staticassertion » Tue, Jul 05, 2022, 8:09 AM UTC

fefe23 wrote: post #47

Why is this interesting to anyone? Haven't we all moved to https by now? Optimizing raw http seems to me like a huge waste of time by now. I say that as someone who has spent years optimizing raw http performance. None of that matters these days.

I wouldn't expect HTTPS to make any difference vs HTTP for long lived connections.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#49

Post by rohith2506 » Tue, Jul 05, 2022, 8:41 AM UTC

fefe23 wrote: post #47

Why is this interesting to anyone? Haven't we all moved to https by now? Optimizing raw http seems to me like a huge waste of time by now. I say that as someone who has spent years optimizing raw http performance. None of that matters these days.

This is particularly interesting in HFT where network latency plays a major role in win ratio

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#50

Post by anonymoushn » Tue, Jul 05, 2022, 8:49 AM UTC

thekozmo wrote: post #46

limoce wrote: post #15

I am not 100% sure that all of the mitigation overhead comes from syscalls, but it stands to reason that a lot of it arises from security hardening in user-to-kernel and kernel-to-user transitions. Will io_uring be also affected by Spectre mitigations given it has eliminated most kernel/user switches? And did anyone do a head-to-head comparison between io_uring and DPDK?

Good point. This is more of a tcp stack comparison between the kernel and userspace. Seastar has a sharded (per core) stack, which is very beneficial when the number of threads is high

You can set up one or many rings per core, but the idea I alluded to elsewhere in this comment section of spending 2 cores to do kernel busy polling and userspace busy polling for a single ring is less useful if your alternative makes good use of all cores.