Why we use the Linux kernel's TCP stack
blog.cloudflare.com
Why we use the Linux kernel's TCP stack
1–10 of 54 posts
Re: Why we use the Linux kernel's TCP stack
#2[deleted]
Re: Why we use the Linux kernel's TCP stack
#3>With this scale of attack the Linux kernel is not enough for us. We must work around it...we added a partial kernel bypass feature to Netmap: that's described in this blog post. With this technique we can offload our anti-DDoS iptables to a very fast userspace process.
What do they mean by "very fast userspace process"? If you're doing the same thing the kernel would be doing, the userspace process should be strictly slower due to context switches. What costs are they saving on here?
Re: Why we use the Linux kernel's TCP stack
#4>With this scale of attack the Linux kernel is not enough for us. We must work around it...we added a partial kernel bypass feature to Netmap: that's described in this blog post. With this technique we can offload our anti-DDoS iptables to a very fast userspace process. What do they mean by "very fast userspace process"? If you're doing the same thing the kernel would be doing, the userspace process should be strictl…
The "very fast userspace process" has direct access to hardware NIC RX queue and is doing busy polling. It uses 100% CPU all the time. The process is faster then kernel because:
- it does less only since it implements only some subset of iptables
- is single threaded, no locks
- doesn't implement TCP
- it's small, no iTLB misses
- the working size set is small, we only deal with couple of packets at a time
- no memory allocations on the hot path (no skb)
- it is doing busy polling, saving the Xus needed for an interrupt context switch
Re: Why we use the Linux kernel's TCP stack
#5Physical limitations of bandwidth on packets per second: https://www.cisco.com/c/en/us/about/security-center/network-...
Why the Linux kernel has a hard time processing more than 1-2M packets per core per second, and patches/improvements for the kernel: https://lwn.net/Articles/629155/
CloudFlare's kernel bypass blog post: https://blog.cloudflare.com/kernel-bypass/
A paper from NTop on doing 10G line rate packet processing, the limitations, and then-current options: http://luca.ntop.org/10g.pdf
NetOptimizer kernel dev blog, where they've maxed out the throughput of a 10G link using the kernel's stack, and details on latency, theoretical maximums and how to test: https://netoptimizer.blogspot.com/search/label/10G
--
The real answer to "Why do we use the Linux kernel's TCP stack?" is that operating systems are designed to help users and programs. They are not designed to be a custom tailored highest-performance cure-all for the highest possible theoretical computing throughput. Using one tcp stack helps users and programs more than each user or program using its own unique stack.
Re: Why we use the Linux kernel's TCP stack
#6I implemented an highly-scalable user-space TCP stack as part of my Master's thesis [1], last year.
One doesn't use an user-space network stack because the Linux's network stack is slow (it's fast), but because it doesn't scale correctly on an high number of CPUs (> 8 cores) [2]. This is because the kernel suffers from some lock contention when accessing the table containing the socket descriptors.
An user-space stack can be significantly faster when the application layer is really simple (e.g. doing some very simple filtering or routing) and when it does not share any mutable state. As soon as your application layer starts sharing mutable states between connections (e.g. like a database), you'll start having contention issues similar to those experienced by the kernel, and you'll not gain anything from using an user-space stack. Very often, applications that can benefit from such as stack can also be scaled easily on multiple machines, and it's usually easier to keep using the Linux's stack and add more servers.
--
[1] https://github.com/RaphaelJ/rusty
[2] https://github.com/RaphaelJ/rusty/blob/master/doc/img/perfor...
Re: Why we use the Linux kernel's TCP stack
#7I implemented an highly-scalable user-space TCP stack as part of my Master's thesis [1], last year. One doesn't use an user-space network stack because the Linux's network stack is slow (it's fast), but because it doesn't scale correctly on an high number of CPUs (> 8 cores) [2]. This is because the kernel suffers from some lock contention when accessing the table containing the socket descriptors. An user-space stac…
> This is because the kernel suffers from some lock contention when accessing the table containing the socket descriptors.
This would indicate the problem is with packet delivery to application. From my experience even packet delivery to "filter" iptables chain is "slow".
But let's assume you are right, can you elaborate? Do you think SO_REUSEPORT on TCP sockets can solve the contention of accept()?
https://lwn.net/Articles/542629/
There are some initiatives improve SO_REUSEPORT CPU affinity, hopefully making it even faster.
Update: I misread. Ok, so "table containing the sockets", but this is just a large hash table, nothing too fancy... aRFS for greater locality?
Re: Why we use the Linux kernel's TCP stack
#8I implemented an highly-scalable user-space TCP stack as part of my Master's thesis [1], last year. One doesn't use an user-space network stack because the Linux's network stack is slow (it's fast), but because it doesn't scale correctly on an high number of CPUs (> 8 cores) [2]. This is because the kernel suffers from some lock contention when accessing the table containing the socket descriptors. An user-space stac…
> This is because the kernel suffers from some lock contention when accessing the table containing the socket descriptors. This would indicate the problem is with packet delivery to application. From my experience even packet delivery to "filter" iptables chain is "slow". But let's assume you are right, can you elaborate? Do you think SO_REUSEPORT on TCP sockets can solve the contention of accept()? https://lwn.net/A…
A single socket can be shared by multiple cores. That means that the kernel must both protect the socket descriptor from concurrent writes, and can't enforce a TCP link to be handled by a defined core (CPU affinity).
Re: Why we use the Linux kernel's TCP stack
#9dpdk and netmap is really only for applications with cooperating I/O processes. This is because the queue of received packets is shared between all process and any of them can delete any packet.
It may not be good for CloudFlare hosting multiple web servers on the same host, but I it could be good for a database or cache server usually run in a LAN with 10Gbit/s network cards.
Re: Why we use the Linux kernel's TCP stack
#10Earlier quoted context omitted.
> This is because the kernel suffers from some lock contention when accessing the table containing the socket descriptors. This would indicate the problem is with packet delivery to application. From my experience even packet delivery to "filter" iptables chain is "slow". But let's assume you are right, can you elaborate? Do you think SO_REUSEPORT on TCP sockets can solve the contention of accept()? https://lwn.net/A…
A single socket can be shared by multiple cores. That means that the kernel must both protect the socket descriptor from concurrent writes, and can't enforce a TCP link to be handled by a defined core (CPU affinity).
> A single socket can be shared by multiple cores
Absolutely, it can. But everybody sane avoids that, pinning worker processes to specific CPU's and not sharing sockets between them. The rule of thumb is that spinlocks on the hot path of socket access become a bunch of no-ops if there is no lock contention.