Realtime metrics using Redis bitmaps
blog.getspool.com
Realtime metrics using Redis bitmaps
1–10 of 21 posts
Re: Realtime metrics using Redis bitmaps
#2How does it compare with just using sets? Usually only a small fraction of your total userbase is online any given day, especially as dead accounts accumulate... wouldn't the bitmaps be inefficiently sparse?
Re: Realtime metrics using Redis bitmaps
#3How does it compare with just using sets? Usually only a small fraction of your total userbase is online any given day, especially as dead accounts accumulate... wouldn't the bitmaps be inefficiently sparse?
There is certainly a tipping point between sets and bitsets. Bitsets are far more efficient for storage and for performing unions or intersections for millions of users. For example, for a typical 10% of user base active daily for mobile apps (http://www.avc.com/a_vc/2011/07/301010.html), bit sets will pay off both in storage and performance.
Obviously, each use case is different and for very sparse case, sets would be a better choice.
Re: Realtime metrics using Redis bitmaps
#4Cool!
Re: Realtime metrics using Redis bitmaps
#5It is very good to see this article and Redis bitmaps exploited, since it is an extremely memory efficient way to store data, and given the encoding, it is extremely fast to also fetch big amount of information this way.
I really suggest to also looking at GETRANGE and SETRANGE operations that allow to access sub-ranges of a large bitmap fetching or setting arbitrary ranges fo bits.
Probably Lua scripting in 2.6 will enable a lot more interesting things combining server side execution and bitmaps. Thanks for sharing this work.
Re: Realtime metrics using Redis bitmaps
#6It is very good to see this article and Redis bitmaps exploited, since it is an extremely memory efficient way to store data, and given the encoding, it is extremely fast to also fetch big amount of information this way. I really suggest to also looking at GETRANGE and SETRANGE operations that allow to access sub-ranges of a large bitmap fetching or setting arbitrary ranges fo bits. Probably Lua scripting in 2.6 will…
Just curious, how does the Lua runtime perform with bitmap operations?
Re: Realtime metrics using Redis bitmaps
#7It is very good to see this article and Redis bitmaps exploited, since it is an extremely memory efficient way to store data, and given the encoding, it is extremely fast to also fetch big amount of information this way. I really suggest to also looking at GETRANGE and SETRANGE operations that allow to access sub-ranges of a large bitmap fetching or setting arbitrary ranges fo bits. Probably Lua scripting in 2.6 will…
Bitmaps surely seem usable in Redis today - how can you do AND, OR, XOR on Redis bitmaps in Lua?
Also, will you create some kind of market for Redis scripts? That seems useful - even if it's just links to Github! ;)
Re: Realtime metrics using Redis bitmaps
#8How does it compare with just using sets? Usually only a small fraction of your total userbase is online any given day, especially as dead accounts accumulate... wouldn't the bitmaps be inefficiently sparse?
You can do clever things with word aligned hybrid bitmaps to heavily compress sparse bitmaps.
Though, if you are storing binary users, 700 million users take less than 90 MB of space (assuming a straight array implementation). Old days wouldn't need to be kept in memory. If you want to think beyond redis, you could keep the active data in redis then flush old data to disk for later querying.
Sounds useful if your users map to a (0, Max] integer representation. Sounds complicated if you use uuids or external vendor IDs for users anywhere (you'd need an intermediate mapping table somewhere).
Re: Realtime metrics using Redis bitmaps
#9It is very good to see this article and Redis bitmaps exploited, since it is an extremely memory efficient way to store data, and given the encoding, it is extremely fast to also fetch big amount of information this way. I really suggest to also looking at GETRANGE and SETRANGE operations that allow to access sub-ranges of a large bitmap fetching or setting arbitrary ranges fo bits. Probably Lua scripting in 2.6 will…
Just curious, how does the Lua runtime perform with bitmap operations?
[deleted]
Re: Realtime metrics using Redis bitmaps
#10It is very good to see this article and Redis bitmaps exploited, since it is an extremely memory efficient way to store data, and given the encoding, it is extremely fast to also fetch big amount of information this way. I really suggest to also looking at GETRANGE and SETRANGE operations that allow to access sub-ranges of a large bitmap fetching or setting arbitrary ranges fo bits. Probably Lua scripting in 2.6 will…
For example, with Lua scripting, you could add custom commands for working with Bloom filters, to do fast approximate set membership testing:
http://en.wikipedia.org/wiki/Bloom_filter
I'm pretty excited about this.