Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use the Linux kernel's TCP stack

blog.cloudflare.com

Post reply on HN
Print view View original

11–20 of 54 posts

Re: Why we use the Linux kernel's TCP stack

#11

Post by joosters »

majke wrote: post #10
raphaelj wrote: post #8

Earlier quoted context omitted.

A single socket can be shared by multiple cores. That means that the kernel must both protect the socket descriptor from concurrent writes, and can't enforce a TCP link to be handled by a defined core (CPU affinity).

> A single socket can be shared by multiple cores Absolutely, it can . But everybody sane avoids that, pinning worker processes to specific CPU's and not sharing sockets between them. The rule of thumb is that spinlocks on the hot path of socket access become a bunch of no-ops if there is no lock contention.

I can't see why pinning is such an obvious choice. The kernel's scheduler may decide that CPU 1 should be woken to handle some new packets, because CPU 2 is busy. If you've pinned the socket to CPU 2, you may be losing out.

I get that there are trade-offs between the two modes: pinning can provide better cache usage, you can avoid some locks (but indirectly make the kernel do the work for you) and so on, but I don't see how you can confidently state that pinning is the 'sane' choice.

In an ideal world, the kernel has a better overview of the network state and CPU state, and therefore is best positioned to decide which CPU should handle each packet.

Re: Why we use the Linux kernel's TCP stack

#12

Post by blastrat »

peterwwillis wrote: post #5

Physical limitations of bandwidth on packets per second: https://www.cisco.com/c/en/us/about/security-center/network-... Why the Linux kernel has a hard time processing more than 1-2M packets per core per second, and patches/improvements for the kernel: https://lwn.net/Articles/629155/ CloudFlare's kernel bypass blog post: https://blog.cloudflare.com/kernel-bypass/ A paper from NTop on doing 10G line rate packet proc…

| The real answer ... is that operating systems are designed to help users and programs. They are not designed to be a custom tailored highest-performance cure-all ... Using one tcp stack helps users and programs more...

I don't use that definition of operating systems, I use the unix virtual machine because it's ubiquitous and provides its same large core set of defined functionalities across many scales and platforms; and it is a particularly genius virtual machine; coding to other virtual machines is more difficult and terribly parochial, decreasing the usefulness of the work.

Much work is done at many layers to the internals of the unix virtual machine to increase its ubiquity, scope, scale, and performance. The more the internal design of the consistent virtual machine can support a custom tailored highest-performance cure-all for the highest possible theoretical computing throughput, the more ubiquitous and useful that virtual machine will be.

Re: Why we use the Linux kernel's TCP stack

#14

Post by felixgallo »

joosters wrote: post #11
majke wrote: post #10

Earlier quoted context omitted.

> A single socket can be shared by multiple cores Absolutely, it can . But everybody sane avoids that, pinning worker processes to specific CPU's and not sharing sockets between them. The rule of thumb is that spinlocks on the hot path of socket access become a bunch of no-ops if there is no lock contention.

I can't see why pinning is such an obvious choice. The kernel's scheduler may decide that CPU 1 should be woken to handle some new packets, because CPU 2 is busy. If you've pinned the socket to CPU 2, you may be losing out. I get that there are trade-offs between the two modes: pinning can provide better cache usage, you can avoid some locks (but indirectly make the kernel do the work for you) and so on, but I don't…

The kernel scheduler has no idea which application thread handles which socket. You can formulate an application level plan and then enforce your will with socket and cpu pinning.

Re: Why we use the Linux kernel's TCP stack

#15

Post by gpderetta »

chmike wrote: post #9

dpdk and netmap is really only for applications with cooperating I/O processes. This is because the queue of received packets is shared between all process and any of them can delete any packet. It may not be good for CloudFlare hosting multiple web servers on the same host, but I it could be good for a database or cache server usually run in a LAN with 10Gbit/s network cards.

Don't modern high performance network cards have multiple tx/rx queues which are virtualizable via IOMMU?

That's a genuine question BTW, I've only a bit of experience with userspace networking with fully cooperating processes.

Re: Why we use the Linux kernel's TCP stack

#16

Post by zeroxfe »

doomrobo wrote: post #3

>With this scale of attack the Linux kernel is not enough for us. We must work around it...we added a partial kernel bypass feature to Netmap: that's described in this blog post. With this technique we can offload our anti-DDoS iptables to a very fast userspace process. What do they mean by "very fast userspace process"? If you're doing the same thing the kernel would be doing, the userspace process should be strictl…

> If you're doing the same thing the kernel would be doing, the userspace process should be strictly slower due to context switches.

Actually, you would cut down on context switches as you ship packets between kernel and user-space. This allows you to do all kinds of intelligent filtering and other header processing that can vastly improve latency in gateway systems (proxies, routers, VM hosts, etc.)

Re: Why we use the Linux kernel's TCP stack

#17

Post by catern »

majke wrote: post #4
doomrobo wrote: post #3

>With this scale of attack the Linux kernel is not enough for us. We must work around it...we added a partial kernel bypass feature to Netmap: that's described in this blog post. With this technique we can offload our anti-DDoS iptables to a very fast userspace process. What do they mean by "very fast userspace process"? If you're doing the same thing the kernel would be doing, the userspace process should be strictl…

The "very fast userspace process" has direct access to hardware NIC RX queue and is doing busy polling. It uses 100% CPU all the time. The process is faster then kernel because: - it does less only since it implements only some subset of iptables - is single threaded, no locks - doesn't implement TCP - it's small, no iTLB misses - the working size set is small, we only deal with couple of packets at a time - no memor…

It seems like the main thing making that process fast is the fact that it's doing polling. But Linux does polling too, when there are enough packets coming in. Likewise, if you really are dropping packets very early in the networking stack, you wouldn't reach the TCP layer and would have a small working set in the kernel too.

So, why not just implement this very fast userspace process as a kernel patch? That seems much easier...

Re: Why we use the Linux kernel's TCP stack

#18

Post by peterwwillis »

blastrat wrote: post #12
peterwwillis wrote: post #5

Physical limitations of bandwidth on packets per second: https://www.cisco.com/c/en/us/about/security-center/network-... Why the Linux kernel has a hard time processing more than 1-2M packets per core per second, and patches/improvements for the kernel: https://lwn.net/Articles/629155/ CloudFlare's kernel bypass blog post: https://blog.cloudflare.com/kernel-bypass/ A paper from NTop on doing 10G line rate packet proc…

| The real answer ... is that operating systems are designed to help users and programs. They are not designed to be a custom tailored highest-performance cure-all ... Using one tcp stack helps users and programs more... I don't use that definition of operating systems, I use the unix virtual machine because it's ubiquitous and provides its same large core set of defined functionalities across many scales and platfor…

There is no such thing. (Was this an attempt at a troll?)

Re: Why we use the Linux kernel's TCP stack

#19

Post by jsnell »

gpderetta wrote: post #15
chmike wrote: post #9

dpdk and netmap is really only for applications with cooperating I/O processes. This is because the queue of received packets is shared between all process and any of them can delete any packet. It may not be good for CloudFlare hosting multiple web servers on the same host, but I it could be good for a database or cache server usually run in a LAN with 10Gbit/s network cards.

Don't modern high performance network cards have multiple tx/rx queues which are virtualizable via IOMMU? That's a genuine question BTW, I've only a bit of experience with userspace networking with fully cooperating processes.

SR-IOV is good for actual virtualization, but it's pretty clumsy for trying to create isolation within a single VM. For example:

- Every VF you create using SR-IOV will need to have a distinct MAC (and thus in practice different IP). But what you'd usually want for this use case is use the same IP for all apps, and do the split by destination port.

- Another consequence of the previos point is that all apps would need to include their own support for ARP, DHCP, etc. Doing it centralized doesn't really work.

- No promiscuous mode (at least on Intel NICs)m you only get traffic directed to one specific MAC address

Now, if you didn't try to use the virtualization support but just use the separate RX/TX queues, with something like the flow director for deciding what traffic gets sent to which queue, you'd get rid of the above problems. But then you end up with the issue that DPDK makes it very hard to have separate applications access the same NIC, even on different queues.

Re: Why we use the Linux kernel's TCP stack

#20

Post by dgemm »

doomrobo wrote: post #3

>With this scale of attack the Linux kernel is not enough for us. We must work around it...we added a partial kernel bypass feature to Netmap: that's described in this blog post. With this technique we can offload our anti-DDoS iptables to a very fast userspace process. What do they mean by "very fast userspace process"? If you're doing the same thing the kernel would be doing, the userspace process should be strictl…

Saving an interrupt per packet. It's far fewer context switches to process packets in a busy loop.
Post reply on HN