41–50 of 51 posts
I am not very familiar with OSes and things at the kernel level, can anyone answer the question in the comments of the post? "How did you measure the time spent in each section (HW, kernel, app)? how did you get such granularity?"
To be clear, 80% of kernel-time in that 1KB write is spent in fsync(), _not_ in the network stack. Network overhead is roughly similar between read & write requests. What Arrakis seems to be able to do is avoid the overhead of write & sync, presumably because it doesn't go through the VFS + filesystem + block IO code paths.
>To be clear, 80% of kernel-time in that 1KB write is spent in fsync(), _not_ in the network stack Are you sure? Of the total 3.36 μs (see Table 1) spent processing each packet in Linux, nearly 70% is spent in the network stack
Earlier quoted context omitted.
>To be clear, 80% of kernel-time in that 1KB write is spent in fsync(), _not_ in the network stack Are you sure? Of the total 3.36 μs (see Table 1) spent processing each packet in Linux, nearly 70% is spent in the network stack
Table 1 is looking specifically at getting a chunk of data off the wire and into the users code. Look at Table 2 for a comparison of redis read/write. Average time for a write is 163 μs - of which 137 μs is spent in fsync(2)
Earlier quoted context omitted.
>To be clear, 80% of kernel-time in that 1KB write is spent in fsync(), _not_ in the network stack Are you sure? Of the total 3.36 μs (see Table 1) spent processing each packet in Linux, nearly 70% is spent in the network stack
Table 1 is looking specifically at getting a chunk of data off the wire and into the users code. Look at Table 2 for a comparison of redis read/write. Average time for a write is 163 μs - of which 137 μs is spent in fsync(2)
This is why high-performance commercial databases do most or all of their I/O management and scheduling in userspace. It is not a new idea; it is much more efficient. However, that means you need to reimplement most of what the kernel does in an optimal way. This is relatively common for closed source software but you almost never see these types of userspace I/O designs in open source, which means OSS designs are of…
The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…
It doesn't make sense for each app server to have an 8gb hash table in their memory nor can they easily be synchronized.
Earlier quoted context omitted.
Having worked on an embedded system that dealt primarily with network I/O (10 Gbps) and hash tables (in the 10 GiB range), I can tell you that network I/O (done right!) and parsing account for MAYBE 1/3 of the total latency of an operation involving a hash table lookup. Memory, like all mass storage, is SLOW once you're not working in cache, and hash tables have no locality. (By "done right!" I mean either batching r…
I've no doubt memory is slow, taking up to 100-300 cycles in NUMA systems. But a single threaded server accessing local memory won't may those costs as much. Are you saying that a few random memory accesses are slower than sending and receiving a packet on two machines? Redis has a great position as a persistent, shareable, data structure server, but replacing in memory hashtables where they work doesn't seem like on…
Of course, you can pipeline memory accesses to some degree, but not as easily as you can aggregate network requests into fewer packets.
I'm certainly not saying the network overhead is free -- it's not! I'm just saying it needn't "eclipse" the hash table lookup itself (as the GP suggested). They're on the same order of magnitude.
The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…
The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…
There's two groups of people who need to squeeze all the work out of every cycle they can get: Embedded programmers (which, despite our massively powerful phone processors, still includes mobile due to power issues), and cloud programmers. Cloud people are totally interested in optimizing everything to within an inch of its life, so it's a valid concern for them that even if they reduce their user space costs to 0 th…
I wonder if anyone has tried it with Solarflare?
Solarflare has a whitepaper on accelerating memcached: http://10gbe.blogspot.com/2014/12/memcached-3x-faster-than-i... I just whipped this up in 5 minutes and didn't do much of the tuning there (e.g. no isolcpus or interrupt changes), but here's a single-client 1024-byte SET redis-benchmark running against localhost with and without TCP Loopback Acceleration... redis 2.8.4 on a dual E5-2630 @ 2.30GHz, card is SFN5122…