In-memory key-value store in C, Go and Python
darkcoding.net
In-memory key-value store in C, Go and Python
1–10 of 58 posts
Re: In-memory key-value store in C, Go and Python
#2In C you are not necessarily "on your own" - you can do something very similar to Python, turning your socket into a file with fdopen(). You can then use functions like fgets() and fread() on it, and stdio will take care of the buffering for you.
Re: In-memory key-value store in C, Go and Python
#3Looking at the go version[1] I am not sure if will work as expected. Maps in go are not thread safe.[2] So it could be the go version is out performing the others do to the lack of synchronisation. There are several ways to fix this in go, the easiest might just be to use a mutex around the accesses to the cache. But it would probably be better to use a readers writers lock.
edit: (that said I am not sure if the C version is thread safe either I haven't read the docs for the hash table he is using.)
edit 2: (looks like the C version is not thread safe either).
[1] https://github.com/grahamking/Key-Value-Polyglot/blob/master...
Re: In-memory key-value store in C, Go and Python
#4Looking at the go version[1] I am not sure if will work as expected. Maps in go are not thread safe.[2] So it could be the go version is out performing the others do to the lack of synchronisation. There are several ways to fix this in go, the easiest might just be to use a mutex around the accesses to the cache. But it would probably be better to use a readers writers lock. edit: (that said I am not sure if the C ve…
You're correct that updating a map from multiple goroutines without synchronizing is unsafe. But the overhead of using a RWMutex (http://weekly.golang.org/pkg/sync/#RWMutex) in this case should be negligible compared to the I/O code.
The big win is that Go allows you to write straightforward concurrent code but under the hood uses high-performance system calls like epoll.
EDIT: Here's a thread-safe version: https://github.com/jbarham/Key-Value-Polyglot/blob/master/me.... Still plenty fast.
Re: In-memory key-value store in C, Go and Python
#5I added an implementation [1] in diesel [2][3], which uses select.epoll (or libev, on non-Linux systems) and got a around 150x speedup [4]. I only repeated the tests a few times (but they were all close) and didn't install the Go compiler so I could test against Go (I'd be interested to see how this stacks up on your machine). Like you say in your post, it's nice to have something wrap up the bother of epoll for you.
[1] https://github.com/wmoss/Key-Value-Polyglot
[2] diesel.io
[3] https://github.com/jamwt/diesel
[4] The first run is against the diesel one
wmoss@wmoss-mba:~/etc/Key-Value-Polyglot$ time python test.py
real 0m0.134s
user 0m0.040s
sys 0m0.020s
wmoss@wmoss-mba:~/etc/Key-Value-Polyglot$ time python test.py
real 0m20.164s
user 0m0.096s
sys 0m0.072s
Re: In-memory key-value store in C, Go and Python
#6I added an implementation [1] in diesel [2][3], which uses select.epoll (or libev, on non-Linux systems) and got a around 150x speedup [4]. I only repeated the tests a few times (but they were all close) and didn't install the Go compiler so I could test against Go (I'd be interested to see how this stacks up on your machine). Like you say in your post, it's nice to have something wrap up the bother of epoll for you.…
also, the diesel one is half the LOC
/shameless plug :-)
(for the record, on my machine the go comparison was 97ms vs. 173ms, so pure python + diesel was 1.78x slower)
Re: In-memory key-value store in C, Go and Python
#7I just happen to start writing a redis clone in python a few weeks ago. I used the ioloop from Tornado to take advantage of epoll (you can also use gevent). Have yet to benchmark it, but I suspect it will bring you closer to the results you see in Go.
Re: In-memory key-value store in C, Go and Python
#8Are we seriously discussing a benchmark that only runs 1000 operations? I don't even understand how it could take 20s to complete in any language on the server side and be correct code. Implement the Redis protocol and use the included redis-benchmark to test your server. On a decent Mac you should be able to hit 500k/s with pipelining and 25k/s without it.
Re: In-memory key-value store in C, Go and Python
#9This comparison is rather unfair on C, where you have chosen to use a low level interface, against Go, where you have chosen to use a high level interface. It is irrelevant that these are the default interfaces - high level interfaces for sockets exist in C. You could even integrate into Nginx.
Re: In-memory key-value store in C, Go and Python
#10Are we seriously discussing a benchmark that only runs 1000 operations? I don't even understand how it could take 20s to complete in any language on the server side and be correct code. Implement the Redis protocol and use the included redis-benchmark to test your server. On a decent Mac you should be able to hit 500k/s with pipelining and 25k/s without it.
I immediately disregard anyone that doesn't show their benchmarks running for any amount of time that would reasonably be used in production.
I have absolutely no problem with letting my stuff fire requests off on 12+ core machines for hours or days on end. And then repeat it. And again.
Production means 24/7 and when I read your less than an hour benchmark on ANY test - well, that's just a not right.