Interesting that Instagram is still (at least partially) on EC2 rather than in Facebook's datacenters.
Instagram: Making the Switch to Cassandra from Redis
21–29 of 29 posts
Re: Instagram: Making the Switch to Cassandra from Redis
#22I think that here the approach is very wise: every time you have a problem involving a big amount of data, Redis may be costly, since it stores data in memory. Sometimes this is unavoidable, for instance I may see Twitter switching from Redis to another in-memory store to cache timelines, but hardly to something using disk. So it is very legit to say, I'm running a number of hosts since my memory requirements are hig…
The cost of VM RAM is bounded below by the cost of the physical RAM used to provide it, unless it's subsidized by costs elsewhere.
Re: Instagram: Making the Switch to Cassandra from Redis
#23The article compares Redis to Cassandra which means comparing apples to oranges for their use case. It would have been meaningful if they explained why they chose Casssandra, instead of for example Hbase.
Exactly the point I was thinking
Re: Instagram: Making the Switch to Cassandra from Redis
#24Interesting that Instagram is still (at least partially) on EC2 rather than in Facebook's datacenters.
I'd tend to think if it ain't broke don't fix it.
Re: Instagram: Making the Switch to Cassandra from Redis
#25I think that here the approach is very wise: every time you have a problem involving a big amount of data, Redis may be costly, since it stores data in memory. Sometimes this is unavoidable, for instance I may see Twitter switching from Redis to another in-memory store to cache timelines, but hardly to something using disk. So it is very legit to say, I'm running a number of hosts since my memory requirements are hig…
I'm not entirely sure what I'd search for so I figure I might as well ask here. When a key won't fit in redis, does redis remove the least active key(s) to make room? And is there a way to persist the removed keys to disk to possibly be retrieved at a later date? That way the most active keys remain in memory while the least active keys are still available.
Re: Instagram: Making the Switch to Cassandra from Redis
#26I think that here the approach is very wise: every time you have a problem involving a big amount of data, Redis may be costly, since it stores data in memory. Sometimes this is unavoidable, for instance I may see Twitter switching from Redis to another in-memory store to cache timelines, but hardly to something using disk. So it is very legit to say, I'm running a number of hosts since my memory requirements are hig…
I'm not entirely sure what I'd search for so I figure I might as well ask here. When a key won't fit in redis, does redis remove the least active key(s) to make room? And is there a way to persist the removed keys to disk to possibly be retrieved at a later date? That way the most active keys remain in memory while the least active keys are still available.
I haven't actually experienced this, someone correct me if I am wrong: you can choose from different policies (LRU, TTL, random) to expire/remove keys, otherwise if you disable maxmemory it will start using the OS virtual memory and slow down.
Redis had a virtual-memory option like you describe at some point, but it was scraped due to poor performance and other issues.
Re: Instagram: Making the Switch to Cassandra from Redis
#27The article compares Redis to Cassandra which means comparing apples to oranges for their use case. It would have been meaningful if they explained why they chose Casssandra, instead of for example Hbase.
According to the article, the interviewee used to work at DataStax (leader in Cassandra) before joining Instagram. I guess this is one of the reason why they first considered Cassandra and it turned out to be very successful. Biased or not, it is always good to have an engineer with deep understanding on the database, rather than just being an ordinary user.
Re: Instagram: Making the Switch to Cassandra from Redis
#28I wonder if being part of Facebook (from which Cassandra originated) has anything to do with the choice for Cassandra over other databases.
Apparently Facebook is not using Cassandra much. For their messaging project they ended up with HBase instead of Cassandra [1]. I don't know if the system replaced the earlier Cassandra system, or complemented it. [1] http://nosql.mypopescu.com/post/1583884165/facebook-the-unde...
It has been replaced entirely
[1] https://www.facebook.com/notes/facebook-engineering/inside-f...
Re: Instagram: Making the Switch to Cassandra from Redis
#29Earlier quoted context omitted.
I'm not entirely sure what I'd search for so I figure I might as well ask here. When a key won't fit in redis, does redis remove the least active key(s) to make room? And is there a way to persist the removed keys to disk to possibly be retrieved at a later date? That way the most active keys remain in memory while the least active keys are still available.
I haven't actually experienced this, someone correct me if I am wrong: you can choose from different policies (LRU, TTL, random) to expire/remove keys, otherwise if you disable maxmemory it will start using the OS virtual memory and slow down. Redis had a virtual-memory option like you describe at some point, but it was scraped due to poor performance and other issues.
Ah LRU and virtual memory are the keywords I was looking for. I couldn't remember! Thanks!