84% of a single-threaded 1KB write in Redis is spent in the kernel • Hacker News Board

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#21

Post by jerf » Tue, Feb 10, 2015, 2:21 PM UTC

yuriks wrote: post #12

The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…

There's two groups of people who need to squeeze all the work out of every cycle they can get: Embedded programmers (which, despite our massively powerful phone processors, still includes mobile due to power issues), and cloud programmers. Cloud people are totally interested in optimizing everything to within an inch of its life, so it's a valid concern for them that even if they reduce their user space costs to 0 they still have limits put on them by the kernel. And cloud people are willing to put a lot of cleverness and work into their core pathways, and will get surprisingly close to the minimum time necessary, so you might be surprised how much they can get done in fractions of a microsecond in their core workload.

You may not have these problems, in which case in addition to "the concept of redis" baffling you, this will seem absurdly performance-sensitive to you. From a desktop programmer or all but the most complicated websites, that is also a sensible perspective. But the niche in which this discussion makes perfect sense is itself pretty large.

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#22

Post by neomantra » Tue, Feb 10, 2015, 2:47 PM UTC

easytiger wrote: post #7

I wonder if anyone has tried it with Solarflare?

Solarflare has a whitepaper on accelerating memcached: http://10gbe.blogspot.com/2014/12/memcached-3x-faster-than-i...

I just whipped this up in 5 minutes and didn't do much of the tuning there (e.g. no isolcpus or interrupt changes), but here's a single-client 1024-byte SET redis-benchmark running against localhost with and without TCP Loopback Acceleration... redis 2.8.4 on a dual E5-2630 @ 2.30GHz, card is SFN5122F but this is all loopback. I'm not claiming anything and just doing it because somebody pondered...

   * plain jane
  /usr/bin/redis-server
  redis-benchmark -t set -q -n 1000000 -d 1024 -c 1
  SET: 21258.96 requests per second
  
   * unaccelerated server, unaccelerated client
   numactl --physcpubind 1,3,5 --preferred 1 /usr/bin/redis-server
   numactl --physcpubind=7,9 --preferred 1  redis-benchmark -t set -q -n 100000 -d 1024 -c 1
  SET: 14293.88 requests per second
  
   * TCP loopback accelerated server, unaccelerated client
  EF_NAME=hn EF_TCP_SERVER_LOOPBACK=2 EF_TCP_CLIENT_LOOPBACK=2 onload -p latency numactl --physcpubind 1,3,5   --preferred 1 /usr/bin/redis-server
  numactl --physcpubind=7,9 --preferred 1  redis-benchmark -t set -q -n   100000 -d 1024 -c 1  
  SET: 25967.28 reque  sts per second

   * TCP loopback accelerated server, accelerated client
  EF_NAME=hn EF_TCP_SERVER_LOOPBACK=2 EF_TCP_CLIENT_LOOPBACK=2 onload -p latency numactl --physcpubind 1,3,5   --preferred 1 /usr/bin/redis-server
  EF_NAME=hn onload -p latency numactl --physcpubind=7 --preferred 1  redis-benchmark -t set -q -n 1000000 -  d 1024 -c 1  
  oo:redis-benchmark[13454]: Sharing OpenOnload 201405-u1 Copyright 2006-2012 Solarflare Communications,   2002-2005 Level 5 Networks [4,hn]
  SET: 96098.41 requests per second

Edit: formatting fixes

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#23

Post by wyager » Tue, Feb 10, 2015, 2:49 PM UTC

When testing https://github.com/wyager/Neks , pipelining made the server something like 30x faster for the exact same reason. Syscalls are tremendously expensive compared to the work of processing a request.

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#24

Post by nteon » Tue, Feb 10, 2015, 2:51 PM UTC

To be clear, 80% of kernel-time in that 1KB write is spent in fsync(), _not_ in the network stack. Network overhead is roughly similar between read & write requests. What Arrakis seems to be able to do is avoid the overhead of write & sync, presumably because it doesn't go through the VFS + filesystem + block IO code paths.

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#25

Post by bwross » Tue, Feb 10, 2015, 2:52 PM UTC

Solutions that pull the TCP stack out of the kernel perform so much better because they're bypassing all the internal bureaucracy that the kernel otherwise performs to make it as easy as possible for userspace applications to use the network without stepping on other applications' toes.

The kernel socket API is designed so that programs have to do as little thinking as possible to get their own personal slice of the shared and noisy network. It provides an easy abstraction, and that requires the kernel do a lot of messy stuff for you:

- When you're using TCP sockets, the kernel makes copies of everything your application writes and holds it in a buffer until its receipt is acknowledged, just in case it needs to resend it when the other side doesn't acknowledge it. If the socket's buffer fills up, your application blocks on I/O until some space is freed.

- It holds ports open in a lingering state long after they're closed just in case it needs to re-transmit the last bytes. This can be disabled, but it's on by default.

- It takes care of all the congestion control for you, but it's tuned for the general case, and as a result there are a lot of edge cases which perform very badly for the problem they're trying to solve. Redis is probably one such edge case.

Of course, all of this is fine and desirable for general applications, but it ends up being problematic if you're trying to solve a problem where performance is the chief concern.

It's tempting to say the problem is that kernel has to do way too much to provide that easy abstraction, but really the problem is that the kernel provides no way around it. You pretty much have the option of using their cushy stream abstraction at the cost of performance, or you use a userspace TCP stack on raw sockets, which requires running as root and disabling TCP in the kernel (otherwise the kernel stomps all over your TCP negotiations[1]).

There are some other transport layer protocols (SCTP, DCCP, etc.), as well as application layer protocols built on UDP, that remove some of the abstractions TCP provides and as a result require less in-kernel bureaucracy, but those solutions don't seem to be very popular or well-supported.

It would be nice if the kernel would provide some lower level system calls that could be selectively used to move parts of TCP into the application (e.g., retaining copies of data in case of re-transmission). Alas, I don't think there's much push for that, because a) it's hard, and b) the current situation is fine for 99% of network applications.

[1] http://jvns.ca/blog/2014/08/12/what-happens-if-you-write-a-t...

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#26

Post by colanderman » Tue, Feb 10, 2015, 3:05 PM UTC

yuriks wrote: post #12

The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…

Having worked on an embedded system that dealt primarily with network I/O (10 Gbps) and hash tables (in the 10 GiB range), I can tell you that network I/O (done right!) and parsing account for MAYBE 1/3 of the total latency of an operation involving a hash table lookup. Memory, like all mass storage, is SLOW once you're not working in cache, and hash tables have no locality.

(By "done right!" I mean either batching requests to/from the kernel, or using a zero-copy userspace solution like DPDK. Clearly in this article network I/O was not done right. Round trips through the kernel ALWAYS will kill performance.)

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#27

Post by moe » Tue, Feb 10, 2015, 3:06 PM UTC

falcolas wrote: post #17

yuriks wrote: post #12

The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…

The value of Redis is never going to be found with a single server. It's going to be found when you use Redis to synchronize the state of multiple servers.

Redis can actually be very useful on a single server, too, as a fast, robust, convenient, potentially shared datastore.

Performance is usually a secondary concern in these use-cases.

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#28

Post by TallGuyShort » Tue, Feb 10, 2015, 3:25 PM UTC

noir-york wrote: post #5

Where exactly in the networking stack is the time being spent? If it is in the IP/TCP layers then moving that to user-space does not, by itself, necessarily reduce latency, it merely shifts it elsewhere. If the latency is due to kernel user land memory copies then that is a different matter.

You can make surprisingly significant gains by writing a TCP/IP stack that is intended for one process only. OSv is a kernel that supports a single process. By running memcached (just as one specific example) on OSv in KVM, you get better performance than by running it on the host. The trick is their network stack is significantly simpler and bypasses the kernel to go straight to the hardware. If you could do the same in user space, that's where you'd get your gains. (But I do agree with you - JUST moving to user-space is rather pointless).

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#29

Post by markbnj » Tue, Feb 10, 2015, 3:33 PM UTC

falcolas wrote: post #17

yuriks wrote: post #12

The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…

The value of Redis is never going to be found with a single server. It's going to be found when you use Redis to synchronize the state of multiple servers.

It makes a great inter-process work queue as well. I don't think you have to stretch your imagination very far to find interesting single server use cases for redis.

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#30

Post by JoeAltmaier » Tue, Feb 10, 2015, 3:45 PM UTC

InfiniBand was conceived around this issue. Additional overhead using kernel I/o includes the user/kernel space switch, copying between user/kernel buffers, and waiting/polling for interrupts. That stuff doesn't get any smaller as networks get faster, so today we're at the point that they dominate the actual hardware I/o time for many network devices.

There's been periodic interest in 'virtual hardware' where the hardware presents multiple interfaces to different users. This way the driver can run in user mode, since there's no need to control/share the hardware registers in the kernel.