How to set up a Redis cluster (preview, not production-ready)
1–10 of 16 posts
Re: How to set up a Redis cluster (preview, not production-ready)
#2The title is a bit misleading :) This is just a preview of Redis Cluster, it's not good to just claiming that I'm working at it, I wanted to show the first stuff we have that users can actually play with (but not use in production environments!).
Re: How to set up a Redis cluster (preview, not production-ready)
#3Hey antirez, both this and 2.4 look really great. Can you talk to some internals of redis-trib? Is it using a cyclical hash on keys approach? What sort of controls are you planning on giving datastore designers?
I was using redis for consuming a multi-gigabyte financial analysis dataset, but eventually moved on because I didn't like using Hiredis on perl or C, so the clustering is very appealing as a way to keep latency down and get bandwidth higher.
Re: How to set up a Redis cluster (preview, not production-ready)
#4The title is a bit misleading :) This is just a preview of Redis Cluster, it's not good to just claiming that I'm working at it, I wanted to show the first stuff we have that users can actually play with (but not use in production environments!).
Ah, sorry, I assumed people would understand that from the post. I added (preview, not production-ready), is that better?
Re: How to set up a Redis cluster (preview, not production-ready)
#5The title is a bit misleading :) This is just a preview of Redis Cluster, it's not good to just claiming that I'm working at it, I wanted to show the first stuff we have that users can actually play with (but not use in production environments!).
Ah, sorry, I assumed people would understand that from the post. I added (preview, not production-ready), is that better?
perfect, thank you.
p.s. I find it very cool that you, working for 10 gen, submitted this article on hacker news. There is still hope in the NoSQL landscape :)
Re: How to set up a Redis cluster (preview, not production-ready)
#6Hey antirez, both this and 2.4 look really great. Can you talk to some internals of redis-trib? Is it using a cyclical hash on keys approach? What sort of controls are you planning on giving datastore designers? I was using redis for consuming a multi-gigabyte financial analysis dataset, but eventually moved on because I didn't like using Hiredis on perl or C, so the clustering is very appealing as a way to keep late…
Hey. All the possible keys are simply divided into 4096 slots. You can assign any part of this slots to any given node, so you can assign things in strange ways, like, node 1 50% of all the nodes, and 10% each to node 2,3,4,5,6.
Redis-trib will simply propose something that makes sense, but you will be able to edit the initial guess of the tool.
About sharding before redis cluster you can do it already but with more problems for sure as resharding will be much simpler with redis cluster (however you can just avoid doing resharding for now, using a lot of nodes from the start, not optimal but works). Not sure what is the problem with hiredis or perl, did not understood this part.
Re: How to set up a Redis cluster (preview, not production-ready)
#7Hey antirez, both this and 2.4 look really great. Can you talk to some internals of redis-trib? Is it using a cyclical hash on keys approach? What sort of controls are you planning on giving datastore designers? I was using redis for consuming a multi-gigabyte financial analysis dataset, but eventually moved on because I didn't like using Hiredis on perl or C, so the clustering is very appealing as a way to keep late…
Hey. All the possible keys are simply divided into 4096 slots. You can assign any part of this slots to any given node, so you can assign things in strange ways, like, node 1 50% of all the nodes, and 10% each to node 2,3,4,5,6. Redis-trib will simply propose something that makes sense, but you will be able to edit the initial guess of the tool. About sharding before redis cluster you can do it already but with more…
hriedis on perl works fine, well, mostly fine. I uncovered at least one small bug which I'll send on at some point.
The issue is that hiredis didn't have a usable python client, and I hated developing in perl, or C. Without hiredis, redis over python is unusably slow for querying 100k+ keys at once, I believe due to python's networking / socket code.
In the end, I ended up moving to pytables for this sort of stuff; it would be nice to use redis for it, but the data access patterns are tuned a little bit more for the web than for a 'do this math on these 3 million data points' kind of use case.
Anyway, I'm excited about redis cluster, I think it fits a huge need for web-scale application development.
Re: How to set up a Redis cluster (preview, not production-ready)
#8Earlier quoted context omitted.
Hey. All the possible keys are simply divided into 4096 slots. You can assign any part of this slots to any given node, so you can assign things in strange ways, like, node 1 50% of all the nodes, and 10% each to node 2,3,4,5,6. Redis-trib will simply propose something that makes sense, but you will be able to edit the initial guess of the tool. About sharding before redis cluster you can do it already but with more…
hriedis on perl works fine, well, mostly fine. I uncovered at least one small bug which I'll send on at some point. The issue is that hiredis didn't have a usable python client, and I hated developing in perl, or C. Without hiredis, redis over python is unusably slow for querying 100k+ keys at once, I believe due to python's networking / socket code. In the end, I ended up moving to pytables for this sort of stuff; i…
Tip: try node/coffeescript and hiredis, you might like it!
Re: How to set up a Redis cluster (preview, not production-ready)
#9Reading about Redis Cluster, I am a bit worried that some re-invention of core erlang features is going on.
If nothing else, an erlang to redis bridge might make for a nice pesudo-cluster...
Re: How to set up a Redis cluster (preview, not production-ready)
#10Reading about Redis Cluster, I am a bit worried that some re-invention of core erlang features is going on. If nothing else, an erlang to redis bridge might make for a nice pesudo-cluster...
Actually, I don't see any reinvention of Erlang's core features here:
- Erlang: a functional language based on actor model that allows to create distributed applications easily as actors don't share state (plus much more that I won't mention here for the sake of brevity). To share state among various Erlang clusters you have to use something like Mnesia (bundled with Erlang stdlib) or Riak (distributed database built using Erlang).
- Redis Cluster: a way to distribute state among multiple Redis (a simple database application) nodes in a manner transparent for the applications using Redis.