Storing hundreds of millions of simple key-value pairs in Redis
instagram-engineering.tumblr.com
Storing hundreds of millions of simple key-value pairs in Redis
1–10 of 55 posts
Re: Storing hundreds of millions of simple key-value pairs in Redis
#2The the code the article links to for zipmap.c (https://github.com/antirez/redis/blob/unstable/src/zipmap.c) is rather literate.
I haven't dug extremely deeply into the sources for many F/OSS projects; the code I'm interested in reading has been inevitably opaque (at least to my inexperience). This particular source file (and maybe the rest of Redis?) is really good. I think I'll be taking many more looks at Redis (code- and usage-wise) in the future.
Re: Storing hundreds of millions of simple key-value pairs in Redis
#3The the code the article links to for zipmap.c ( https://github.com/antirez/redis/blob/unstable/src/zipmap.c ) is rather literate. I haven't dug extremely deeply into the sources for many F/OSS projects; the code I'm interested in reading has been inevitably opaque (at least to my inexperience). This particular source file (and maybe the rest of Redis?) is really good. I think I'll be taking many more looks at Redis…
Back when we were getting started with Redis, the readability / concise nature of the project was one of the things that most excited me about it (here's what it looked like around then: https://github.com/antirez/redis/tree/0b420168b485d0a9c4b66d...)
Re: Storing hundreds of millions of simple key-value pairs in Redis
#4Best of all, lookups in hashes are still O(1), making them very quick.
How quick is "very quick"? I was hoping to see some performance benchmarks, not just memory usage benchmarks.
Re: Storing hundreds of millions of simple key-value pairs in Redis
#5I love companies like Instagram who take the time to share their scaling issues and experiences.
Re: Storing hundreds of millions of simple key-value pairs in Redis
#6Best of all, lookups in hashes are still O(1), making them very quick. How quick is "very quick"? I was hoping to see some performance benchmarks, not just memory usage benchmarks.
Redis includes a benchmark utility (http://redis.io/topics/benchmarks). I ran it on my MacBook Pro a couple of weeks ago and got north of 50,000 req/s with a bunch of other crap running. I don't think the utility has any tests for hashes, though.
Re: Storing hundreds of millions of simple key-value pairs in Redis
#7You can find similar examples -- ruby code for the "exact" implementation as described by Instagram -- of this and other redis, memory optimization(s) at the Redis site: http://redis.io/topics/memory-optimization
Re: Storing hundreds of millions of simple key-value pairs in Redis
#8Best of all, lookups in hashes are still O(1), making them very quick. How quick is "very quick"? I was hoping to see some performance benchmarks, not just memory usage benchmarks.
Redis includes a benchmark utility ( http://redis.io/topics/benchmarks ). I ran it on my MacBook Pro a couple of weeks ago and got north of 50,000 req/s with a bunch of other crap running. I don't think the utility has any tests for hashes, though.
Oh, I've seen other redis benchmarks. But it's always nice to see more -- that way when someone wants to guess how fast their problem set will run on their hardware they will have a more similar benchmark to compare it against.
Re: Storing hundreds of millions of simple key-value pairs in Redis
#91 million pair using 16 MB is about 16 bytes per pair, which is perfectly fine but nothing impressive.
The dataset is static, so a simple naive solution would be to create a big array sorted by key. Assuming both photo and user IDs use 4 bytes each, this would result in about 2GB of data. Then use binary search to lookup values.
However, if we really want to reduce the size, we could build a finite state machine from the dataset (maybe reverse the values to increase the level of shared suffixes) which should reduce the size by an order of magnitude.
Re: Storing hundreds of millions of simple key-value pairs in Redis
#10The the code the article links to for zipmap.c ( https://github.com/antirez/redis/blob/unstable/src/zipmap.c ) is rather literate. I haven't dug extremely deeply into the sources for many F/OSS projects; the code I'm interested in reading has been inevitably opaque (at least to my inexperience). This particular source file (and maybe the rest of Redis?) is really good. I think I'll be taking many more looks at Redis…
Back when we were getting started with Redis, the readability / concise nature of the project was one of the things that most excited me about it (here's what it looked like around then: https://github.com/antirez/redis/tree/0b420168b485d0a9c4b66d... )
I'm curious, what in particular makes you say that code is "readable"?
Coding style varies dramatically from person to person, and I don't mean this as a criticism of antirez, but any code which doesn't have at minimum a one-line comment before each function explaining its purpose immediately fails the "readability" test for me. Obviously this isn't a problem for you, so I'm curious to hear what your tastes are.