Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

84% of a single-threaded 1KB write in Redis is spent in the kernel

blog.nullspace.io

Post reply on HN
Print view View original

31–40 of 51 posts

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#31

Post by bkeroack »

falcolas wrote: post #17
yuriks wrote: post #12

The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…

The value of Redis is never going to be found with a single server. It's going to be found when you use Redis to synchronize the state of multiple servers.

There are better, more reliable ways to do this than redis. NSQ for example, in a reactive (or log-based) pattern.

Regardless, your goal should be reducing shared state as much as possible, since synchronizing it among distributed systems is a Hard Problem(TM).

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#33

Post by hyc_symas »

antirez wrote: post #15
jdub wrote: post #3

Earlier quoted context omitted.

Point (2) makes me think of LMDB. Have you looked into it much? I wonder if it would be an interesting storage substrate for a threaded Redis.

Point "2" better applies when the storage substrate is memory and operations are O(1) or logarithmic, because in that case, the time to serve the query is comparable small compared to the time needed to process the reply and send back the response. With an on-disk storage, I would go for a classic on-disk database setup where different queries are served by different threads.

LMDB beats all other "classic" on-disk databases for read performance. It also happens to beat all other in-memory systems for read performance too, since its reads require no locks.

http://symas.com/mdb/memcache/ http://symas.com/mdb/inmem/ http://symas.com/mdb/ondisk/

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#34

Post by jandrewrogers »

This is why high-performance commercial databases do most or all of their I/O management and scheduling in userspace. It is not a new idea; it is much more efficient. However, that means you need to reimplement most of what the kernel does in an optimal way.

This is relatively common for closed source software but you almost never see these types of userspace I/O designs in open source, which means OSS designs are often leaving integer factors worth of efficiency and performance on the table. For some use cases, companies are very successful selling into this efficiency and performance gap with closed source.

Part of the lack of open source is that these designs are not portable, due to OS dependencies and sometimes hardware dependencies if a design is hardcore. I think this is a partial copout; a Linux-only database engine would address the vast majority of real-world deployments. A bigger reason is that the design and implementation of these kinds of userspace kernels is a very high skill and low level art that, frankly, is way outside the expertise of most open source software contributors. For databases in particular, more often than not even the basic design elements of the software are naively done (e.g. MongoDB) and that is lower hanging fruit.

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#35

Post by maccard »

jerf wrote: post #21
yuriks wrote: post #12

The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…

There's two groups of people who need to squeeze all the work out of every cycle they can get: Embedded programmers (which, despite our massively powerful phone processors, still includes mobile due to power issues), and cloud programmers. Cloud people are totally interested in optimizing everything to within an inch of its life, so it's a valid concern for them that even if they reduce their user space costs to 0 th…

Game developers also belong to that group of people who squeeze work out of every cycle. Less than 18 months ago, two of the most commonly used devices contained: 512MB dedicated RAM (with 10MB of VRAM) (xbox 360) and 256MB RAM and 256MB VRAM (PS3)

alongside very aged processors. The hardware was almost 10 years old in both cases. Even the current gen aren't particularly powerful, coming in at 8GB ram with a 1.75GHz processor, and a GPU comparable to a 3-4 year old PC for the xbox one, and 1.6GHz processors, 8GB ram and a slightly beefier GPU in the case of the PS4.

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#36

Post by MichaelGG »

colanderman wrote: post #26
yuriks wrote: post #12

The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…

Having worked on an embedded system that dealt primarily with network I/O (10 Gbps) and hash tables (in the 10 GiB range), I can tell you that network I/O (done right!) and parsing account for MAYBE 1/3 of the total latency of an operation involving a hash table lookup. Memory, like all mass storage, is SLOW once you're not working in cache, and hash tables have no locality. (By "done right!" I mean either batching r…

I've no doubt memory is slow, taking up to 100-300 cycles in NUMA systems. But a single threaded server accessing local memory won't may those costs as much. Are you saying that a few random memory accesses are slower than sending and receiving a packet on two machines?

Redis has a great position as a persistent, shareable, data structure server, but replacing in memory hashtables where they work doesn't seem like one of those cases.

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#37

Post by hurin »

nteon wrote: post #24

To be clear, 80% of kernel-time in that 1KB write is spent in fsync(), _not_ in the network stack. Network overhead is roughly similar between read & write requests. What Arrakis seems to be able to do is avoid the overhead of write & sync, presumably because it doesn't go through the VFS + filesystem + block IO code paths.

>To be clear, 80% of kernel-time in that 1KB write is spent in fsync(), _not_ in the network stack

Are you sure?

Of the total 3.36 μs (see Table 1) spent processing each packet in Linux, nearly 70% is spent in the network stack

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#38

Post by bch »

falcolas wrote: post #17
yuriks wrote: post #12

The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…

The value of Redis is never going to be found with a single server. It's going to be found when you use Redis to synchronize the state of multiple servers.

I've used Redis as bounded temporal storage for "log data" -- specifically, on an occasionally crashy Solaris box w/o capabilities to store a days worth of tcpdump data, I would pipe it to Redis w/ timeouts and have a 15-minute sliding window of data -- so when the host bit the dust, I had what I needed, stored safely. That was valuable.

Re: 84% of a single-threaded 1KB write in Redis is spent in the kernel

#40

Post by bherms »

moe wrote: post #27
falcolas wrote: post #17

Earlier quoted context omitted.

The value of Redis is never going to be found with a single server. It's going to be found when you use Redis to synchronize the state of multiple servers.

Redis can actually be very useful on a single server, too, as a fast, robust, convenient, potentially shared datastore. Performance is usually a secondary concern in these use-cases.

Yep, for data stores (caches, etc), Redis is awesome and super fast, though not as time-tested as memcached.

And all of the Redis set/list operations are super valuable. You just need to be careful once you start relying on Redis at scale for things you take for granted when you start playing around with it... For example: zunionstores on lots of large sorted sets. "O(N)+O(M log(M)) with N being the sum of the sizes of the input sorted sets, and M being the number of elements in the resulting sorted set." When you start off using it, it's awesome and super fast, but before you know it, the blocking, single-threaded architecture will crash and burn if your data scales up. Luckily we have clustering now :)

Post reply on HN