nitpick:
"The values are almost inconsequential since THEIR just numbers" should be either THEY ARE or THEY'RE
Snappy Dashboards with Redis
11–20 of 46 posts
Re: Snappy Dashboards with Redis
#12This doesn't make any sense to me. Why store the values in Redis and use Ruby to do simple math? If you're using Postgres already, just store the stats in Postgres and use an aggregate or window function to grab the stats. (And collect those stats with triggers, in the first place.) If you're using Mongo already, just grab your stats with a map reduce query.
It looks like one of the things they're counting is clicks, so they could potentially have some pretty large datasets.
I don't know how well Mongo's map-reduce works, but in Postgres, COUNT(star) [1] does not perform well for very large tables (e.g. 100 million rows). You wouldn't want to be doing a COUNT(star) once per minute for each customer that had their dashboard open on a plasma screen.
Of course, there are other solutions to that problem: generate the counts on some more feasible schedule and cache them; have a read replica used for analytics queries; shard by customer and have no large customers.
I don't know whether their scale strictly requires the Redis solution, but in any case there are situations where it's not as simple as "throw it in Postgres and use an aggregate function".
[1] "star" instead of an asterisk to avoid HN thinking I'm trying to write in italics.
Re: Snappy Dashboards with Redis
#13This is exactly what I'm building http://www.instahero.com to solve. You don't have to build your own infrastructure, just write the relevant bit of code (or select a template) and you have a dashboard.
Re: Snappy Dashboards with Redis
#14This doesn't make any sense to me. Why store the values in Redis and use Ruby to do simple math? If you're using Postgres already, just store the stats in Postgres and use an aggregate or window function to grab the stats. (And collect those stats with triggers, in the first place.) If you're using Mongo already, just grab your stats with a map reduce query.
It looks like one of the things they're counting is clicks, so they could potentially have some pretty large datasets. I don't know how well Mongo's map-reduce works, but in Postgres, COUNT(star) [1] does not perform well for very large tables (e.g. 100 million rows). You wouldn't want to be doing a COUNT(star) once per minute for each customer that had their dashboard open on a plasma screen. Of course, there are ot…
The article said they could rebuild the count if they needed to. So something in regards to each click is being stored. If you were using postgres you'd just setup a trigger on that table to increment the click values (stored in another table) as appropriate. No aggregate function needed.
Re: Snappy Dashboards with Redis
#15This doesn't make any sense to me. Why store the values in Redis and use Ruby to do simple math? If you're using Postgres already, just store the stats in Postgres and use an aggregate or window function to grab the stats. (And collect those stats with triggers, in the first place.) If you're using Mongo already, just grab your stats with a map reduce query.
if it doesn't make sense to you, and you suggest using triggers (!) and mongodb map reduce (!) to get stats, then you are probably working at very small scale. There are a number of problems at small-medium to large scale, at which your mentioned solutions trivially fall apart. That's when you start using tools like Redis -- as scalpels to fix general-case solutions.
Re: Snappy Dashboards with Redis
#16Earlier quoted context omitted.
It looks like one of the things they're counting is clicks, so they could potentially have some pretty large datasets. I don't know how well Mongo's map-reduce works, but in Postgres, COUNT(star) [1] does not perform well for very large tables (e.g. 100 million rows). You wouldn't want to be doing a COUNT(star) once per minute for each customer that had their dashboard open on a plasma screen. Of course, there are ot…
The article said they could rebuild the count if they needed to. So something in regards to each click is being stored. If you were using postgres you'd just setup a trigger on that table to increment the click values (stored in another table) as appropriate. No aggregate function needed.
Certainly you can implement this same pattern without Redis. Triggers in Postgres would be a reasonable way to do it. I didn't say you can't do this in Postgres, I said you can't do it with COUNT().
It does indeed sound like they're storing every click: that's precisely why using aggregate functions would be expensive.
Re: Snappy Dashboards with Redis
#17I use Fnordmetric - Ruby/Redis dashboard. Works great.
Re: Snappy Dashboards with Redis
#18This works great, one thing I've started doing is caching those mget's in memcached. Redis is fast, but it's also single threaded and depending on how you're using it can become cpu bound, causing timeout errors while busy redis instance handle lots of write/reads... so similar to mysql - i've started guarding multiple redis reads with single memcache get... feels crazy, but maybe correct?
I'm surprised that Redis cannot handle your traffic. What are you throwing at it? Or are you on EC2, with frequent BGSAVEs or AOF enabled on an EBS volume?
rackspacecloud - no AOF only BGSAVEs currently my save frequency is:
save 1900 1
save 1300 10
save 160 10000
I should add, I also use redis to handle some pretty large calculations... zrange's for distance calculations intersections and so on... I did some benchmarking and found that at least in ruby (1.9.3), this was more efficient to load up redis for the sets and sorting, fewer GC hits and faster sorting intersecting... I'm thinking it might be good if for these one off frequent compute tasks, I run on a redis instance that has no save, especially considering I expire/delete the keys immediately after running my calculations.
Re: Snappy Dashboards with Redis
#19Earlier quoted context omitted.
The article said they could rebuild the count if they needed to. So something in regards to each click is being stored. If you were using postgres you'd just setup a trigger on that table to increment the click values (stored in another table) as appropriate. No aggregate function needed.
Certainly you can implement this same pattern without Redis. Triggers in Postgres would be a reasonable way to do it. I didn't say you can't do this in Postgres, I said you can't do it with COUNT(). It does indeed sound like they're storing every click: that's precisely why using aggregate functions would be expensive.
You absolutely can do it without COUNT. Just increment a counter value, the same way they're doing it in Redis.
Re: Snappy Dashboards with Redis
#20Earlier quoted context omitted.
I'm surprised that Redis cannot handle your traffic. What are you throwing at it? Or are you on EC2, with frequent BGSAVEs or AOF enabled on an EBS volume?
rackspacecloud - no AOF only BGSAVEs currently my save frequency is: save 1900 1 save 1300 10 save 160 10000 Perhaps there's a better way for me to tune this? Most of the data I store in redis is temporal so I don't mind losing it, but I do store stats for usage of features for reporting in my admin dashboard similar to how this article describes and that stuff i would like to keep around but if i lost a few hours or…
Tried running the redis benchmark app to see how your figures compare to other published stats?