In-memory key-value store in C, Go and Python • Hacker News Board

Re: In-memory key-value store in C, Go and Python

#41

Post by drivebyacct2 » Wed, Mar 21, 2012, 4:17 PM UTC

apaprocki wrote: post #35

Not everyone has this problem, but Go only works on a portion of platform configurations that are available in the real world. C and Python, OTOH, are available pretty much anywhere. Redis builds on say, Solaris, with no problem because the project is written in C and it is trivial to add the needed calls. A KV store written in Go can't support Solaris because Go itself would need to support Solaris first. Years of t…

Uh where is python available that Go is not and how portable is that C code:

Re: In-memory key-value store in C, Go and Python

#42

Post by drivebyacct2 » Wed, Mar 21, 2012, 4:20 PM UTC

pjmlp wrote: post #27

alexchamberlain wrote: post #9

This comparison is rather unfair on C, where you have chosen to use a low level interface, against Go, where you have chosen to use a high level interface. It is irrelevant that these are the default interfaces - high level interfaces for sockets exist in C. You could even integrate into Nginx.

This is typical in the Go community. Usually the supposedly Go advantages are presented in a way, as if the same are not present in other languages.

The author is up front about there existing more optimal and performant designs for python and C but remarks that this is a fast, easy, naive implementation. Who here doesn't appreciate those qualities or who here hasn't worked on a team where there life would have been easier if that one guy had a little bit harder time screwing things up?

Re: In-memory key-value store in C, Go and Python

#43

Post by willbmoss » Wed, Mar 21, 2012, 4:33 PM UTC

chmike wrote: post #32

willbmoss wrote: post #5

I added an implementation [1] in diesel [2][3], which uses select.epoll (or libev, on non-Linux systems) and got a around 150x speedup [4]. I only repeated the tests a few times (but they were all close) and didn't install the Go compiler so I could test against Go (I'd be interested to see how this stacks up on your machine). Like you say in your post, it's nice to have something wrap up the bother of epoll for you.…

What is the difference between diesel and gevent ? Note that gevent 1.0 uses libev.

In a nutshell, gevent monkey patches the socket library, whereas diesel doesn't. This means that you can use any (previously) blocking libraries with gevent, whereas, in diesel you have to write them again. The upside of the rewriting is that it creates a more coherent (and opinionated) ecosystem.

Re: In-memory key-value store in C, Go and Python

#44

Post by jamwt » Wed, Mar 21, 2012, 5:12 PM UTC

chmike wrote: post #32

willbmoss wrote: post #5

I added an implementation [1] in diesel [2][3], which uses select.epoll (or libev, on non-Linux systems) and got a around 150x speedup [4]. I only repeated the tests a few times (but they were all close) and didn't install the Go compiler so I could test against Go (I'd be interested to see how this stacks up on your machine). Like you say in your post, it's nice to have something wrap up the bother of epoll for you.…

What is the difference between diesel and gevent ? Note that gevent 1.0 uses libev.

http://diesel.io/faq

Re: In-memory key-value store in C, Go and Python

#45

Post by ootachi » Wed, Mar 21, 2012, 5:21 PM UTC

drivebyacct2 wrote: post #42

pjmlp wrote: post #27

Earlier quoted context omitted.

This is typical in the Go community. Usually the supposedly Go advantages are presented in a way, as if the same are not present in other languages.

The author is up front about there existing more optimal and performant designs for python and C but remarks that this is a fast, easy, naive implementation. Who here doesn't appreciate those qualities or who here hasn't worked on a team where there life would have been easier if that one guy had a little bit harder time screwing things up?

Careful with that argument. The fast, easy, naïve implementation in Go is quite dangerous (not even memory safe), because he forgot to lock the map. Go made it easy to write an incorrect implementation of a key-value store...

Re: In-memory key-value store in C, Go and Python

#46

Post by apaprocki » Wed, Mar 21, 2012, 5:44 PM UTC

drivebyacct2 wrote: post #41

apaprocki wrote: post #35

Not everyone has this problem, but Go only works on a portion of platform configurations that are available in the real world. C and Python, OTOH, are available pretty much anywhere. Redis builds on say, Solaris, with no problem because the project is written in C and it is trivial to add the needed calls. A KV store written in Go can't support Solaris because Go itself would need to support Solaris first. Years of t…

Uh where is python available that Go is not and how portable is that C code:

I am using Python (as well as Perl, OCaml, etc) on Solaris/Sparc, AIX/POWER7, HP-UX/IA64 on a daily basis. As for C code, If you just want to use redis as an example, I built and installed it on Solaris/Sparc with no incident. This is the case for most C POSIX compliant apps.

Re: In-memory key-value store in C, Go and Python

#47

Post by tptacek » Wed, Mar 21, 2012, 6:15 PM UTC

drivebyacct2 wrote: post #41

apaprocki wrote: post #35

Not everyone has this problem, but Go only works on a portion of platform configurations that are available in the real world. C and Python, OTOH, are available pretty much anywhere. Redis builds on say, Solaris, with no problem because the project is written in C and it is trivial to add the needed calls. A KV store written in Go can't support Solaris because Go itself would need to support Solaris first. Years of t…

Uh where is python available that Go is not and how portable is that C code:

He just said where.

Re: In-memory key-value store in C, Go and Python

#48

Post by tptacek » Wed, Mar 21, 2012, 6:16 PM UTC

apaprocki wrote: post #46

drivebyacct2 wrote: post #41

Earlier quoted context omitted.

Uh where is python available that Go is not and how portable is that C code:

I am using Python (as well as Perl, OCaml, etc) on Solaris/Sparc, AIX/POWER7, HP-UX/IA64 on a daily basis. As for C code, If you just want to use redis as an example, I built and installed it on Solaris/Sparc with no incident. This is the case for most C POSIX compliant apps.

For a long time, Redis didn't even have a configure script; you just typed "make" and it worked. It's scrupulously portable code.

Re: In-memory key-value store in C, Go and Python

#49

Post by apaprocki » Wed, Mar 21, 2012, 6:29 PM UTC

tptacek wrote: post #48

apaprocki wrote: post #46

Earlier quoted context omitted.

I am using Python (as well as Perl, OCaml, etc) on Solaris/Sparc, AIX/POWER7, HP-UX/IA64 on a daily basis. As for C code, If you just want to use redis as an example, I built and installed it on Solaris/Sparc with no incident. This is the case for most C POSIX compliant apps.

For a long time, Redis didn't even have a configure script; you just typed "make" and it worked. It's scrupulously portable code.

Aside: I noticed that redis/Solaris used select() instead of the fancy new "event completion framework"[1] in Solaris 10. I figured maybe the new API would be faster and I could contribute that back, so I ported to both that framework, poll(), and /dev/poll (I feel the event API just wraps /dev/poll due to the perf #s I saw, but it isn't clear) and it turned out that select() is actually faster than any of them, which struck me as a bit odd.

[1]: http://developers.sun.com/solaris/articles/event_completion....

Re: In-memory key-value store in C, Go and Python

#50

Post by SamReidHughes » Wed, Mar 21, 2012, 10:12 PM UTC

It's pretty silly to stress-test a C or Go-written program using a client written in Python. It's very easy for the stress client to be the bottleneck in benchmarking memcached implementations. Even when the client is written in C.