11–20 of 51 posts
This does tie you in to a specific network vendor but I can see the argument for moving more commodity networking hardware in this direction too.
So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead look ridiculous, when it'd be very much acceptable if your server did more actual work.
The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…
Totally... this is why pipelining makes Redis 10x faster, less syscalls. Basically to make Redis much faster we need to work to three different related things: 1) Less kernel friction. 2) Threaded I/O, this is the part worth threading, with a global lock to execute queries so you don't get crazy with concurrency and complex data structures. Memcached did it right. 3) Pipelining: better client support for pipelining s…
Point (2) makes me think of LMDB. Have you looked into it much? I wonder if it would be an interesting storage substrate for a threaded Redis.
The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…
This is why techniques like kernel bypass are used in high throughout or low latency systems like finance. Things like http://www.openonload.org/ This does tie you in to a specific network vendor but I can see the argument for moving more commodity networking hardware in this direction too.
The concept of redis has always baffled me. A hash table is a very fast data structure. As soon as you put that in a dedicated server, the cost of the actual lookup is instantly eclipsed by the need to parse a text protocol and do network I/O to communicate with the client. So I'd be willing to say that the problem here isn't that the kernel stack is slow per-se, but that workload is too small as to make the overhead…
Also, saying that using Redis is slower than using a local hash table is a truism. There are myriad reasons why using a local, in-memory data structure is not viable: scalability and persistence, for example. It's like saying "I don't need a database, I can store everything in a local variable."
Totally... this is why pipelining makes Redis 10x faster, less syscalls. Basically to make Redis much faster we need to work to three different related things: 1) Less kernel friction. 2) Threaded I/O, this is the part worth threading, with a global lock to execute queries so you don't get crazy with concurrency and complex data structures. Memcached did it right. 3) Pipelining: better client support for pipelining s…