Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Storing hundreds of millions of simple key-value pairs in Redis

instagram-engineering.tumblr.com

Post reply on HN
Print view View original

51–55 of 55 posts

Re: Storing hundreds of millions of simple key-value pairs in Redis

#51

Post by ralph »

qeorge wrote: post #36
cperciva wrote: post #35

Earlier quoted context omitted.

Very readable for something so dense. Nice work! Thanks! you could return 1 on success, and keep returning something falsy on failure I come from an OS background, so to me 0 is success and non-zero is failure. It doesn't really matter which convention a project uses as long as it's consistent, so I documented this in my /STYLE file: "In general, functions should return (int)(-1) or NULL to indicate error."

Gotcha. I'm not an OS guy, and didn't know that convention. Thanks! Dumb question though: If rehash can only return -1 or 0, won't if(rehash(H)) always fail?

The reason for the convention is that there's often only one form of success, so 0 suffices, but there may be many causes of failure.

Re: Storing hundreds of millions of simple key-value pairs in Redis

#52

Post by catwell »

vijaydev wrote: post #50
catwell wrote: post #49

Earlier quoted context omitted.

You're probably wrong. Look at the comment above "stretches" here: https://github.com/binarylogic/authlogic/blob/master/lib/aut... This is crypto-related code in of the most popular Rails authentication gems. And this is one among many examples. I'm sure you've already come across something like that more than once. I trust code rather than comments. At least it doesn't lie.

Fixed :-) https://github.com/binarylogic/authlogic/commit/f52a699bbc9e...

Incorrectly, see my comment on your link. But thanks for trying :) If you want my PoV on this, the right way to fix it is delete the comment.

Re: Storing hundreds of millions of simple key-value pairs in Redis

#53

Post by ctoestreich »

pedigree wrote: post #16

Why use clear text numbers? Most of the time, you're going to be using large numbers, so binary pack them as save more space. i had the same issue, normal storage was 1.1gb of space, HSET down to 200mb and binary packing every integer down dbl() bought it right down to 163mb of memory (32bit instance). For that 163mb, I was slicing a md5 of the field for the hset prefix, packing that and then using the remainer as th…

I would like to see some example code or a gist of what you did. I played around with custom number packing and using the hashset in redis inspired by the original thread topic and your comment. You can find the post here: http://www.christianoestreich.com/2011/11/redis-hashsets-per...

Re: Storing hundreds of millions of simple key-value pairs in Redis

#54

Post by ctoestreich »

zellyn wrote: post #23

Why use the whole Media ID as the key within the bucket, rather than just the last three digits?

I think they were trying to keep the hash sizes to ~1000 elements each key. At a 3 digit key you would be increasing the hashes to 300k elements each. You should augment the scripts and see if that has an impact on the performance, would be curious to see.

Re: Storing hundreds of millions of simple key-value pairs in Redis

#55

Post by mkjones »

gorset wrote: post #26
maurits wrote: post #18

Earlier quoted context omitted.

build a finite state machine from the dataset Could you elaborate on that? Or provide a link for the novice?

Sure, I implemented the paper "How to squeeze a lexicon" http://www.n3labs.com/pdf/lexicon-squeeze.pdf with a few tweaks a while back. It uses a simple fixed format for all nodes, which works very well for datasets which contains many common prefixes and suffixes. For example, a polish dictionary containing 1.3 million strings is compressed down to 726KB. One of the thing I'm using it for is in a database for geopip…

Isn't this basically a trie?
Post reply on HN