AOL, Meet Riak

#1

Post by franckcuny » Mon, Mar 21, 2011, 4:49 PM UTC

Re: AOL, Meet Riak

#2

Post by billybob » Mon, Mar 21, 2011, 6:45 PM UTC

We're using Riak to store forms that can have varying schemas as JSON. That and the no-single-point-of-failure were two of the big draws.

Re: AOL, Meet Riak

#3

Post by siculars » Mon, Mar 21, 2011, 8:23 PM UTC

Looks like this guy built a zookeeper lite on top of riak_core. That looks interesting. For those that don't know, the "riak" nosql product is actually comprised of many submodules that each handle a certain portion of the overall application. Riak_core is the module that handles the coordination of distributing work amongst many nodes in a single cluster.

Re: AOL, Meet Riak

#4

Post by foobarbazetc » Tue, Mar 22, 2011, 3:16 AM UTC

Serious question: how do you guys get around the slowness of Riak? I've tried deploying it to like 5-6 nodes, and I max out at like 5000 GET/s with multiple clients/threads/whatever for a single key on giant ass machines with plenty of CPU and RAM.

Re: AOL, Meet Riak

#5

Post by rzezeski » Tue, Mar 22, 2011, 1:50 PM UTC

foobarbazetc wrote: post #4

Serious question: how do you guys get around the slowness of Riak? I've tried deploying it to like 5-6 nodes, and I max out at like 5000 GET/s with multiple clients/threads/whatever for a single key on giant ass machines with plenty of CPU and RAM.

You are performing a GET for the same key each time?

Don't quote me on this but I took a quick glance at the code and it looks like all gets are [currently] serialized at the vnode level. Since your key will always hash to the same vnodes (increasing machine count doesn't help) it means that request will pile up on the vnodes' mailbox.

I also see an asynchronous get defined in the vnode but don't see anyway to use it.

That said, if this isn't an artificial use case then I'd probably throw a short-lived cache in front of Riak for situations like this. I mean, if this were a more traditional database, you wouldn't run the same query 5000 times if you know the answer isn't likely to change.

Re: AOL, Meet Riak

#6

Post by megaman821 » Tue, Mar 22, 2011, 2:05 PM UTC

foobarbazetc wrote: post #4

Serious question: how do you guys get around the slowness of Riak? I've tried deploying it to like 5-6 nodes, and I max out at like 5000 GET/s with multiple clients/threads/whatever for a single key on giant ass machines with plenty of CPU and RAM.

I haven't used Riak but just in general, if I had that kind of load I would use Varnish in front or use a lighter protocol than http like protobufs.

Re: AOL, Meet Riak

#7

Post by foobarbazetc » Wed, Mar 23, 2011, 10:56 PM UTC

rzezeski wrote: post #5

foobarbazetc wrote: post #4

Serious question: how do you guys get around the slowness of Riak? I've tried deploying it to like 5-6 nodes, and I max out at like 5000 GET/s with multiple clients/threads/whatever for a single key on giant ass machines with plenty of CPU and RAM.

You are performing a GET for the same key each time? Don't quote me on this but I took a quick glance at the code and it looks like all gets are [currently] serialized at the vnode level. Since your key will always hash to the same vnodes (increasing machine count doesn't help) it means that request will pile up on the vnodes' mailbox. I also see an asynchronous get defined in the vnode but don't see anyway to use it…

Thanks -- the serialization causing the bottleneck makes sense. I'll try with more keys. :)