Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Realtime metrics using Redis bitmaps

blog.getspool.com

Post reply on HN
Print view View original

11–20 of 21 posts

Re: Realtime metrics using Redis bitmaps

#11

Post by rubyorchard »

antirez wrote: post #5

It is very good to see this article and Redis bitmaps exploited, since it is an extremely memory efficient way to store data, and given the encoding, it is extremely fast to also fetch big amount of information this way. I really suggest to also looking at GETRANGE and SETRANGE operations that allow to access sub-ranges of a large bitmap fetching or setting arbitrary ranges fo bits. Probably Lua scripting in 2.6 will…

Great point and looking forward to Lua support in 2.6. GETRANGE and SETRANGE operations also allow buffered writes at the collection time because setting bit to 1 is an idempotent operation. They are particularly attractive in sharded environment where one can perform:

  WATCH
  bits = GETRANGE ...
  bits = bits | buffered_bits
  MULTI
  SETRANGE .... bits
  EXEC

Re: Realtime metrics using Redis bitmaps

#12

Post by wahnfrieden »

An honest question: what use are realtime metrics if you can't act on them in realtime? If it takes a day or more to gather enough data to make a decision and react to it with code or otherwise, then anything more than daily metrics seem like a distraction. I suppose it can be useful if you have some automated systems in place or if you're using it for alarms.

Re: Realtime metrics using Redis bitmaps

#13

Post by jphackworth »

wahnfrieden wrote: post #12

An honest question: what use are realtime metrics if you can't act on them in realtime? If it takes a day or more to gather enough data to make a decision and react to it with code or otherwise, then anything more than daily metrics seem like a distraction. I suppose it can be useful if you have some automated systems in place or if you're using it for alarms.

If you're pushing changes to production many times a day, you can use realtime metrics to notice if anything goes awry, and know when you should roll back. It won't give you the same nuances as an A/B test over a month, but it's a great way to do a quick double check to keep things moving quickly while avoiding disaster.

Re: Realtime metrics using Redis bitmaps

#17

Post by darklajid »

yread wrote: post #15

How do they know that user ids are contiguous?

The same way the length of a meter is known. By defining it. They are measuring visitors, they can assign any numbers they like to those.

In my (limited) use cases for redis which interfaced with external information like that I usually had a mapping table anyway, to get something like

users:nameFromExternalSource:myIncrementalId

foo:myId:ThatUsersFoo

bar:myId:ThatUsersBar

Note again that I'm no expert on redis. There might be problems with that approach that I don't know about - but for me this worked out quite well and seemed to save memory vs any foo:externalIdOrUsername:dataHere name scheme.

Re: Realtime metrics using Redis bitmaps

#18

Post by pbrumm »

antirez wrote: post #5

It is very good to see this article and Redis bitmaps exploited, since it is an extremely memory efficient way to store data, and given the encoding, it is extremely fast to also fetch big amount of information this way. I really suggest to also looking at GETRANGE and SETRANGE operations that allow to access sub-ranges of a large bitmap fetching or setting arbitrary ranges fo bits. Probably Lua scripting in 2.6 will…

It would be really cool if redis supported the ability to count the high bits in a string as a native command. No need to pull a 16k string down to client

Although it is another command in a growing list of them. maybe something for lua.

Re: Realtime metrics using Redis bitmaps

#20

Post by pjscott »

pbrumm wrote: post #18
antirez wrote: post #5

It is very good to see this article and Redis bitmaps exploited, since it is an extremely memory efficient way to store data, and given the encoding, it is extremely fast to also fetch big amount of information this way. I really suggest to also looking at GETRANGE and SETRANGE operations that allow to access sub-ranges of a large bitmap fetching or setting arbitrary ranges fo bits. Probably Lua scripting in 2.6 will…

It would be really cool if redis supported the ability to count the high bits in a string as a native command. No need to pull a 16k string down to client Although it is another command in a growing list of them. maybe something for lua.

This is something that should probably be implemented natively, for speed. Preferably with GCC's __builtin_popcount intrinsic, if it's available.
Post reply on HN