84% of a single-threaded 1KB write in Redis is spent in the kernel
blog.nullspace.io
84% of a single-threaded 1KB write in Redis is spent in the kernel
1–10 of 51 posts
Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel
#2Totally... this is why pipelining makes Redis 10x faster, less syscalls.
Basically to make Redis much faster we need to work to three different related things:
1) Less kernel friction.
2) Threaded I/O, this is the part worth threading, with a global lock to execute queries so you don't get crazy with concurrency and complex data structures. Memcached did it right.
3) Pipelining: better client support for pipelining so that it's easy to tell the client in what order I need my replies, and unrelated replies can be glued together easily.
Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel
#3Totally... this is why pipelining makes Redis 10x faster, less syscalls. Basically to make Redis much faster we need to work to three different related things: 1) Less kernel friction. 2) Threaded I/O, this is the part worth threading, with a global lock to execute queries so you don't get crazy with concurrency and complex data structures. Memcached did it right. 3) Pipelining: better client support for pipelining s…
Point (2) makes me think of LMDB. Have you looked into it much? I wonder if it would be an interesting storage substrate for a threaded Redis.
Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel
#4Totally... this is why pipelining makes Redis 10x faster, less syscalls. Basically to make Redis much faster we need to work to three different related things: 1) Less kernel friction. 2) Threaded I/O, this is the part worth threading, with a global lock to execute queries so you don't get crazy with concurrency and complex data structures. Memcached did it right. 3) Pipelining: better client support for pipelining s…
(Author here) Continuing our thread from Twitter, one of the things I wanted to say here is that things like pipelining and threading are good, but there is still improvement to be made at the kernel I/O layer. If we can reduce this latency there could be very compelling cost incentives especially at the datacenter level, and it might simplify application logic quite a bit.
This is one of the big reasons I like the paper I mention in this post.
Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel
#5Where exactly in the networking stack is the time being spent?
If it is in the IP/TCP layers then moving that to user-space does not, by itself, necessarily reduce latency, it merely shifts it elsewhere. If the latency is due to kernel user land memory copies then that is a different matter.
Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel
#6Where exactly in the networking stack is the time being spent? If it is in the IP/TCP layers then moving that to user-space does not, by itself, necessarily reduce latency, it merely shifts it elsewhere. If the latency is due to kernel user land memory copies then that is a different matter.
The point of moving it to user-space is that you can source a lot of the jobs of the TCP/IP stack to hardware directly. In some cases this dramatically speeds up your I/O.
Consider, for example that the dominant costs are things like demultiplexing and security checks. If you choose to implement multiplexing with virtual network cards then you get true 0-copy multiplexing, which is much faster than the software equivalent. And many of the security checks can be eliminated by using some combination of packet filters and logical disks. (The security BTW seems to be one big difference from RDMA, which might be an alternative, but I'm not really an expert.)
Some things can't be sourced to the hardware, like naming and access control. But that's fine.
(NB, I'm not arguing for this paper's position necessarily, I just thought it was interesting, and the motivation was good enough to start me thinking about how I might get around the kernel.)
Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel
#7I wonder if anyone has tried it with Solarflare?
Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel
#8This article from last month is relevant:
Improving Linux networking performance
https://lwn.net/Articles/629155/
HN discussion: https://news.ycombinator.com/item?id=8931431
Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel
#9Where exactly in the networking stack is the time being spent? If it is in the IP/TCP layers then moving that to user-space does not, by itself, necessarily reduce latency, it merely shifts it elsewhere. If the latency is due to kernel user land memory copies then that is a different matter.
I'd like to know that too e.g. even if you just have 1 iptables rule, that don't match your traffic, you'll still take a rather big performance hit in the network stack for everything.
Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel
#10Where exactly in the networking stack is the time being spent? If it is in the IP/TCP layers then moving that to user-space does not, by itself, necessarily reduce latency, it merely shifts it elsewhere. If the latency is due to kernel user land memory copies then that is a different matter.
The point of moving it to user-space is that you can source a lot of the jobs of the TCP/IP stack to hardware directly. In some cases this dramatically speeds up your I/O. Consider, for example that the dominant costs are things like demultiplexing and security checks. If you choose to implement multiplexing with virtual network cards then you get true 0-copy multiplexing, which is much faster than the software equiv…
The life of a data frame has become pretty complicated. Along the way it may pass through one or more layers of virtualization, one or more layers of network-specific mangling like iptables doing filtering and NAT, and a combo of the two in OS-based virtualized networks connecting VMs and containers both intra- and inter-host.
Pushing more of the stack into hardware is probably a good idea for single-tenant datacenters that can deploy a lot of e.g. Redis appliances, but those of us just renting capacity in the cloud are going to suffer from Amdahl's Law if you can only accelerate the part of the system adjacent to real hardware NICs.