Linux Kernel vs. DPDK: HTTP Performance Showdown • Hacker News Board

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#61

Post by mgaunard » Tue, Jul 05, 2022, 12:23 PM UTC

JoshTriplett wrote: post #8

evgpbfhnr wrote: post #5

At the point you've gotten syscall overhead is definitely going to be a big thing (even without spectre mitigations enabled) -- I'd be very curious to see how far a similar io_uring benchmark would get you. It supports IOPOLL (polling of the socket) and SQPOLL (kernel side polling of the request queue) so hopefully the fact that application driving it is in another thread wouldn't slow it too much... With multi-shot…

I would love to see an io_uring comparison as well; while it's a substantial amount of work to port an existing framework to io_uring, at the point where you're considering DPDK, io_uring seems relatively small by comparison.

I personally started a new framework and I went for io_uring for simplicity, that is already giving most of what I need -- asynchronous I/O with no context switching.

DPDK is huge and inflexible, it does a lot of things which I'd rather be in control of myself and I think it's easier to just do my own userspace vfio.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#62

Post by salmo » Tue, Jul 05, 2022, 12:41 PM UTC

hedora wrote: post #20

salmo wrote: post #4

Earlier quoted context omitted.

For regulatory compliance, that’s still not acceptable because it opens the door to cross user data access or privilege escalation. If someone exploited a process with a dedicated unprivileged user, had legit limited access, or got in a container on a physical, they might be able to leverage it for the forces of evil. There’s really no such practical thing as single user Linux. If you’re running a network exposed app…

I suspect you are conflating security regulations for Unix users with regulations targeting users of the system. Why would a regulatory framework care if a Linux box running one process was vulnerable to attacks that involve switching UIDs? Converse, why would that same regulatory framework not care if users of that network service were able to impersonate each other / access each others’ data?

Most of the controls are about auditability and data access.

But the control frameworks are silly sometimes. Then add in that they’re enforced by 3rd party auditor consultants looking for any reason to drag it out.

And yeah, I tried to get this past them for a old singleton system to avoid having to buy a bigger non-standard server.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#63

Post by salmo » Tue, Jul 05, 2022, 12:45 PM UTC

throwawaylinux wrote: post #10

salmo wrote: post #4

Earlier quoted context omitted.

For regulatory compliance, that’s still not acceptable because it opens the door to cross user data access or privilege escalation. If someone exploited a process with a dedicated unprivileged user, had legit limited access, or got in a container on a physical, they might be able to leverage it for the forces of evil. There’s really no such practical thing as single user Linux. If you’re running a network exposed app…

Which regulations?

The worst is the fake regulation that is PCI. But I tried in a SOX audit to avoid buying new gear.

I hate auditors. Making sense doesn’t matter. Their interpretation of the control does. I do have a playbook of “how I meet X but not the way you want me to”, but lost that one. Probably spent more $ arguing than the HW cost.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#64

Post by maxgio92 » Tue, Jul 05, 2022, 1:44 PM UTC

Thank you, very exhaustive and interesting. A note: the link to bftrace script is broken.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#65

Post by gjulianm » Tue, Jul 05, 2022, 1:47 PM UTC

fefe23 wrote: post #47

Why is this interesting to anyone? Haven't we all moved to https by now? Optimizing raw http seems to me like a huge waste of time by now. I say that as someone who has spent years optimizing raw http performance. None of that matters these days.

For me the interest comes from seeing the speed boost between regular kernel/optimized kernel/DPDK. What you put behind the RX layer doesn't really matter, but it's good to see numbers and things to do when your RX system isn't giving you enough throughput.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#66

Post by pclmulqdq » Tue, Jul 05, 2022, 3:44 PM UTC

medawsonjr wrote: post #18

pclmulqdq wrote: post #3

This was a fascinating read and the kernel does quite nicely in comparison - 66% of DPDK performance is amazing. That said, the article completely nails the performance advantage: DPDK doesn't do a lot of stuff that the kernel does. That stuff takes time. If I recall correctly, DPDK abstractions themselves cost a bit of NIC performance, so it might be interesting to see a comparison including a raw NIC-specific kerne…

Or better yet, Mellanox VMA since it's open source (unlike Solarflare OpenOnload) and the NICs are far less expensive.

I have found OpenOnload to be easier to use than VMA, although I think you can go a bit faster with Mellanox NICs.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#67

Post by menaerus » Tue, Jul 05, 2022, 4:14 PM UTC

thekozmo wrote: post #41

BeefWellington wrote: post #29

Earlier quoted context omitted.

> Yea, it is definitely a fake HTTP server which I acknowledge in the article It's not actually an HTTP server though... For these purposes, it's essentially no more useful than netcat dumping out a preconfigured text file. Titling it "HTTP Performance showdown" is doubly bad here since there's no real-world (or even moderately synthetic) HTTP requests happening; you just always get the same static set of data for ev…

ScyllaDB uses Seastar as an engine and the DynamoDB compatible API use HTTP parsing, so this use case is real. Of course the DB has much more to do than this benchmark with a static http reply but Scylla also uses many more core in the server, thus it is close to real life. We do use the kernel's tcp stack, due to all of its features and also since we don't have capacity for a deeper analysis. Some K/V workloads are…

Few questions if you will, it's an interesting work and I figure you're on ScyllaDB team?

1. Is 5s experiment with 1s warmup really a representative workload? How about running for several minutes or tens of minutes? Do you observe the same result?

2. How about 256 connections on 16 vCPUs creating contention against each other and therefore skewing the experiment results? Aren't they competing for the same resources against each other?

3. Are the experiment results reproducible on different machines (at first use the same and then similar SW+HW configurations)?

4. How many times is experiment (benchmark) repeated and what about the statistical significance of the observed results? How do you make sure to understand that what you're observing, and hence drawing a conclusion out of it in the end, is really what you thought you were measuring?

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#68

Post by dclusin » Tue, Jul 05, 2022, 5:13 PM UTC

grive wrote: post #51

dclusin wrote: post #26

Earlier quoted context omitted.

It’s typically done when end to end latency is more important than full protocol compliance and hardening against myriad types of attackers. High frequency trading is typically where the applications where you see these sorts of implementations being really compelling. For these types of applications there are proprietary implementations that you can buy from vendors that are more suited to latency sensitive applicat…

FPGAs are falling out of favor, ASICs are sufficient to implement most network processing. Funnily, one of the biggest DPDK feature is an API to program smartNICs exactly in that way.

Fair enough. My Knowledge of what’s on the bleeding edge and mass adoption in algo trading is fairly dated. I haven’t been adjacent to that industry in over 5 years. Even back then there was turnkey ASICS you could buy that would implement the network plus whatever protocols you used to talk to the exchange.

I think the neatest thing that sticks out to me is how logging in implementations of low latency trading applications was essentially pushed to the network. Basically they just have copper or fiber taps between each node in their system and it gets sent to another box that aggregates the traffic and processes all the packets to provide trace level logging through their entire system. Even these have solutions you can buy from a number of vendors.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#69

Post by thekozmo » Tue, Jul 05, 2022, 6:59 PM UTC

menaerus wrote: post #67

thekozmo wrote: post #41

Earlier quoted context omitted.

ScyllaDB uses Seastar as an engine and the DynamoDB compatible API use HTTP parsing, so this use case is real. Of course the DB has much more to do than this benchmark with a static http reply but Scylla also uses many more core in the server, thus it is close to real life. We do use the kernel's tcp stack, due to all of its features and also since we don't have capacity for a deeper analysis. Some K/V workloads are…

Few questions if you will, it's an interesting work and I figure you're on ScyllaDB team? 1. Is 5s experiment with 1s warmup really a representative workload? How about running for several minutes or tens of minutes? Do you observe the same result? 2. How about 256 connections on 16 vCPUs creating contention against each other and therefore skewing the experiment results? Aren't they competing for the same resources…

Am ScyllaDB but Marc did completely independent work. The client vcpus don't matter that much, the experiment compares the server side, the client shouldn't suck. When we test ScyllaDB or other DBs, we run benchmarks for hours and days. This is just a stateless, static http daemon, so short timing is reasonable.

The whole intent is to make it a learning experience, if you wish to reproduce, try it yourself. It's aligned with past measurements of ours and also with former Linux optimizations by Marc.

Re: Linux Kernel vs. DPDK: HTTP Performance Showdown

#70

Post by Matthias247 » Tue, Jul 05, 2022, 11:36 PM UTC

gjulianm wrote: post #58

touisteur wrote: post #43

Earlier quoted context omitted.

And now you're in AF_XDP territory, which is an alternative to dpdk, a tentative to normalise kernel 'bypass' for network operations.

I wouldn't say XDP is an alternative to DPDK. XDP severely limits what you can do as you insert code inside of the kernel. DPDK exposes the NIC to regular user code so you can do what you want. XDP is useful when you want to modify what the kernel does with packets before they go to their destination application, DPDK when you want to do whatever with the packets without sending them to any other application.

That's why the parent mentions AF_XDP. AF_XDP allows to get raw packets into userspace, bypassing the kernel stack. There users can run e.g. a custom TCP stack to process the packets. That's similar to what can be done with DPDK.

And both tools allow to export a "regular NIC" to the user. With AF_XDP one would probably just run the AF_XDP socket in a way which intercepts packets for a certain port, and leave all other traffic alone. That would then flow through the regular kernel stack, and the network interface would continue to show up as without XDP.