Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

In-memory key-value store in C, Go and Python

darkcoding.net

Post reply on HN
Print view View original

31–40 of 58 posts

Re: In-memory key-value store in C, Go and Python

#31

Post by codeape »

caf wrote: post #20
comex wrote: post #11

Just running (a single instance of) test.py as a benchmark does not make sense. epoll is optimized for efficiently handling large numbers of sockets, but here there is only one socket. There is no reason epoll should be faster at blocking socket I/O than blocking socket I/O; if it is, I blame the kernel. (Incidentally, here on OS X where there is no epoll, all the solutions performed pretty terribly - a few seconds f…

Yes, the explanation given does not seem to pass the sniff test. If all the wall-time is being taken in recv(), there's no reason why using epoll() would be any improvement.

The reason the python version is slow is (I believe) that the code is very inefficient. It uses socket.sendall() instead of sockfile.write/flush. Using sockfile.write/flush speeds it up from 50 requests/second to 7k requests/second on my machine.

Re: In-memory key-value store in C, Go and Python

#32

Post by chmike »

willbmoss wrote: post #5

I added an implementation [1] in diesel [2][3], which uses select.epoll (or libev, on non-Linux systems) and got a around 150x speedup [4]. I only repeated the tests a few times (but they were all close) and didn't install the Go compiler so I could test against Go (I'd be interested to see how this stacks up on your machine). Like you say in your post, it's nice to have something wrap up the bother of epoll for you.…

What is the difference between diesel and gevent ? Note that gevent 1.0 uses libev.

Re: In-memory key-value store in C, Go and Python

#33

Post by yummyfajitas »

antirez wrote: post #18
willvarfar wrote: post #16

Earlier quoted context omitted.

As the author points out in the article, the neat thing is that Go does this for you. That was the major takeaway.

This is the default programming model in Go, but it's the same for Node.js, and you can use an easy-to-use event library in Ruby and Python as well. A key-value store is system programming. It should be carefully designed, it's not real-world that you accept the default I/O model of the language you are going to use. p.s. in the case of C it's hard to argue what is the default I/O model. It supports threads, fork, mu…

In the real world, defaults and simple-to-use libraries (i.e. not twisted) matter. Just because you are doing system programming doesn't mean you necessarily have the time/skill to carefully optimize everything.

System's programming isn't just writing a KV store used by millions. Sometimes it's writing a dedicated calculation server used in a single company. Carefully optimizing your I/O model might take longer than doing the project itself.

Re: In-memory key-value store in C, Go and Python

#34

Post by sausagefeet »

axbx wrote: post #10
spullara wrote: post #8

Are we seriously discussing a benchmark that only runs 1000 operations? I don't even understand how it could take 20s to complete in any language on the server side and be correct code. Implement the Redis protocol and use the included redis-benchmark to test your server. On a decent Mac you should be able to hit 500k/s with pipelining and 25k/s without it.

I immediately disregard anyone that doesn't show their benchmarks running for any amount of time that would reasonably be used in production. I have absolutely no problem with letting my stuff fire requests off on 12+ core machines for hours or days on end. And then repeat it. And again. Production means 24/7 and when I read your less than an hour benchmark on ANY test - well, that's just a not right.

Totally, you just don't see some things after a long time. I haven't read the code but people mentioned the author is using malloc in a lot of places, you wont' see ill effects from memory fragmentation with a quick benchmark, it could take hours or days to see how large of holes you have in your memory space.

Re: In-memory key-value store in C, Go and Python

#35

Post by apaprocki »

Not everyone has this problem, but Go only works on a portion of platform configurations that are available in the real world.

C and Python, OTOH, are available pretty much anywhere. Redis builds on say, Solaris, with no problem because the project is written in C and it is trivial to add the needed calls. A KV store written in Go can't support Solaris because Go itself would need to support Solaris first.

Years of tooling centered around C (e.g., autoconf/automake) is what makes most C programs cross-platform out of the box with little or no OS-specific code if you are sticking to POSIX. Until the same ecosystem develops around any new language, authors realize that choice of language alone can immediately limit their cross-platform capabilities.

Re: In-memory key-value store in C, Go and Python

#36

Post by teejae »

jbarham wrote: post #4
timtadh wrote: post #3

Looking at the go version[1] I am not sure if will work as expected. Maps in go are not thread safe.[2] So it could be the go version is out performing the others do to the lack of synchronisation. There are several ways to fix this in go, the easiest might just be to use a mutex around the accesses to the cache. But it would probably be better to use a readers writers lock. edit: (that said I am not sure if the C ve…

You're correct that updating a map from multiple goroutines without synchronizing is unsafe. But the overhead of using a RWMutex ( http://weekly.golang.org/pkg/sync/#RWMutex ) in this case should be negligible compared to the I/O code. The big win is that Go allows you to write straightforward concurrent code but under the hood uses high-performance system calls like epoll. EDIT: Here's a thread-safe version: https:/…

FYI, In the Go version, the lock can be added more simply, without the rw:

// Synchronize map access between multiple goroutines. type cache struct { m map[string]string sync.RWMutex }

Then, you can just use:

cache.Lock/Unlock/RLock/RUnlock directly. It's clearer. The beauty of Go's mixins.

Re: In-memory key-value store in C, Go and Python

#37

Post by ootachi »

yummyfajitas wrote: post #33
antirez wrote: post #18

Earlier quoted context omitted.

This is the default programming model in Go, but it's the same for Node.js, and you can use an easy-to-use event library in Ruby and Python as well. A key-value store is system programming. It should be carefully designed, it's not real-world that you accept the default I/O model of the language you are going to use. p.s. in the case of C it's hard to argue what is the default I/O model. It supports threads, fork, mu…

In the real world, defaults and simple-to-use libraries (i.e. not twisted) matter. Just because you are doing system programming doesn't mean you necessarily have the time/skill to carefully optimize everything. System's programming isn't just writing a KV store used by millions. Sometimes it's writing a dedicated calculation server used in a single company. Carefully optimizing your I/O model might take longer than…

There is no default in C, beyond stdio. The C specification makes no reference to any I/O beyond stdio.

The author was being unfair to C. Use epoll or, better yet, libuv. libuv is a simple-to-use library.

Re: In-memory key-value store in C, Go and Python

#38

Post by ootachi »

jbarham wrote: post #4
timtadh wrote: post #3

Looking at the go version[1] I am not sure if will work as expected. Maps in go are not thread safe.[2] So it could be the go version is out performing the others do to the lack of synchronisation. There are several ways to fix this in go, the easiest might just be to use a mutex around the accesses to the cache. But it would probably be better to use a readers writers lock. edit: (that said I am not sure if the C ve…

You're correct that updating a map from multiple goroutines without synchronizing is unsafe. But the overhead of using a RWMutex ( http://weekly.golang.org/pkg/sync/#RWMutex ) in this case should be negligible compared to the I/O code. The big win is that Go allows you to write straightforward concurrent code but under the hood uses high-performance system calls like epoll. EDIT: Here's a thread-safe version: https:/…

So much for CSP, if you're using threads and shared state.

Re: In-memory key-value store in C, Go and Python

#39

Post by ootachi »

sausagefeet wrote: post #34
axbx wrote: post #10

Earlier quoted context omitted.

I immediately disregard anyone that doesn't show their benchmarks running for any amount of time that would reasonably be used in production. I have absolutely no problem with letting my stuff fire requests off on 12+ core machines for hours or days on end. And then repeat it. And again. Production means 24/7 and when I read your less than an hour benchmark on ANY test - well, that's just a not right.

Totally, you just don't see some things after a long time. I haven't read the code but people mentioned the author is using malloc in a lot of places, you wont' see ill effects from memory fragmentation with a quick benchmark, it could take hours or days to see how large of holes you have in your memory space.

And in Go, they aren't running long enough to see the GC kick in. I suspect the overhead of GC will be significant without tuning.

Re: In-memory key-value store in C, Go and Python

#40

Post by jlouis »

samwilliams wrote: post #14

Interesting! I was actually thinking about writing something very similar as an erlang C node just a couple of days a go. I noted that the overhead for storing a mnesia table of 5 million rows of 3 integers was huge - it would take up 1.6gb in memory! If you know the size of the struct, it should pretty easy to make a fast lookup system (assuming the keys are sequential) too. I wonder if I could wrap this instead...

Use an ETS table and mark it as compressed :)
Post reply on HN