Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving persistent data out of Redis

githubengineering.com

Post reply on HN
Print view View original

21–30 of 77 posts

Re: Moving persistent data out of Redis

#21

Post by jhgg »

We've recently had to move away from redis for persistent data storage at work too - opting instead to write a service layer ontop of cassandra for storing data.

Redis was tremendous in our journey up there - but one of the shortcomings is that it isn't as easy to scale-up as cassandra is if you haven't designed your system to scale-up on redis from when it was built (which we didn't) - instead of re-architecting for a redis-cluster setup, we decided to move the component to a clustered microservice written in go, that sits as a memory-cache & write buffer infront of cassandra for hot, highly mutated data.

Would anyone be interested in a blog post about our struggles & journey?

Re: Moving persistent data out of Redis

#22

Post by meritt »

jhgg wrote: post #21

We've recently had to move away from redis for persistent data storage at work too - opting instead to write a service layer ontop of cassandra for storing data. Redis was tremendous in our journey up there - but one of the shortcomings is that it isn't as easy to scale-up as cassandra is if you haven't designed your system to scale-up on redis from when it was built (which we didn't) - instead of re-architecting for…

I'd be interested in seeing that, also exactly what bottlenecks were you running into? CPU? I/O? Memory?

Re: Moving persistent data out of Redis

#23

Post by jhgg »

meritt wrote: post #22
jhgg wrote: post #21

We've recently had to move away from redis for persistent data storage at work too - opting instead to write a service layer ontop of cassandra for storing data. Redis was tremendous in our journey up there - but one of the shortcomings is that it isn't as easy to scale-up as cassandra is if you haven't designed your system to scale-up on redis from when it was built (which we didn't) - instead of re-architecting for…

I'd be interested in seeing that, also exactly what bottlenecks were you running into? CPU? I/O? Memory?

Memory & CPU. We knew we'd be running into both eventually - without adding additional redis instances (memory growing way faster than CPU usage in this specific scenario).

When it was initially built, it was basically a bunch of redis lua scripts to handle updating the data - on redis configured in master/slave managed by sentinels.

Given the nature of the data too, only the data for active users would be hot - but users that were inactive would have stuck around in memory needlessly. Our new system keeps only the hot set of users in memory. We also built it to transparently migrate users from redis to cassandra when they were accessed.

Re: Moving persistent data out of Redis

#24

Post by activatedgeek »

Can anybody here help me understand why many teams are using MySQL as a KV store? (Uber did it recently, so assuming many others probably did it too, network effect)

I personally love MySQL. Just want to understand what makes MySQL a great KV store as opposed to more seemingly specialized systems like Redis?

Re: Moving persistent data out of Redis

#25

Post by antirez »

virmundi wrote: post #15
antirez wrote: post #8

Earlier quoted context omitted.

That's a valid reason indeed. Also technological consolidation, that is, if I can do everything with a single DB / language / ... I always tend to use a single thing.

Watch how far you generalize that view. I realize that "I can do everything with a single...language" can make "everything" mean anything. I had a coworker that helped a middle eastern country with security software. He went full Javascript on it: Node, Mongo, etc. It worked. JS did everything. While he didn't go into details, afterwards he thought it was a bad idea. A great learning experience on when to define "eve…

Yes I totally agree that like most "rules on software" there is the need to be judicious enough to know when to follow the rule is not a good idea... However I more often see the contrary, of adding a multitude of systems together without very strong reasons.

Re: Moving persistent data out of Redis

#26

Post by antirez »

rthrfrd wrote: post #10
antirez wrote: post #3

Wow I'm learning today that Github used Redis for persistent data, now that they moved away :-) Anyway very happy that Redis helped to run such an important site. From the blog post it looks like that for certain things to move away from Redis was hard even if they are very skilled with MySQL, this is a good thing from the POV of Redis since it means that Redis allows to model certain things easily. However they want…

FWIW we went through a very similar process to that documented here by Github (~3 months ago). It was entirely due to operational reasons and nothing to with shortcomings in Redis itself. MySQL was the master record for 99% of our data while Redis was the master record for the other 1% (as it happens it was also a kind of activity stream). Having the single 'master' reference for our data reduced complexity to a degr…

Yes, this helps and totally makes sense to me. Thanks. I would do the same... In this case however it looks like there were certain high volume writes that could be handled in a simpler manner with Redis, however it is totally possible that while this looks like an important use case, it accounted for a small percentage of all the data, so we are back to the consolidation thing of moving everything to a single system that is in general a good idea.

Re: Moving persistent data out of Redis

#27

Post by sacheendra »

parthdesai wrote: post #12
rthrfrd wrote: post #10

Earlier quoted context omitted.

FWIW we went through a very similar process to that documented here by Github (~3 months ago). It was entirely due to operational reasons and nothing to with shortcomings in Redis itself. MySQL was the master record for 99% of our data while Redis was the master record for the other 1% (as it happens it was also a kind of activity stream). Having the single 'master' reference for our data reduced complexity to a degr…

just asking for some info, but how do you make sure that multiple of your db systems are in sync (specifically interested in MySql and elasticsearch)? Hope it's alright to ask you that.

If you need any kind of consistency guarantee, I think you would need to use some kind of distributed transactions.

If its not, you could tail the MySQL log and have a process making the same changes to elasticsearch. The elasticsearch may lag behind if there are problems.

Re: Moving persistent data out of Redis

#28

Post by virmundi »

ohstopitu wrote: post #17
virmundi wrote: post #15

Earlier quoted context omitted.

Watch how far you generalize that view. I realize that "I can do everything with a single...language" can make "everything" mean anything. I had a coworker that helped a middle eastern country with security software. He went full Javascript on it: Node, Mongo, etc. It worked. JS did everything. While he didn't go into details, afterwards he thought it was a bad idea. A great learning experience on when to define "eve…

just wondering why Mongo is considered a part of "JS". Is it because of the MEAN stack?

Part of it is JSON as the storage format. Another part is its Node driver. The whole API fit it well. It understood async programming. It felt JS-like. The input and output were JSON instances. Finally, yea, from what I know of him "web scale" did play part in the decision. Oh those heady days.

Re: Moving persistent data out of Redis

#29

Post by Sanddancer »

activatedgeek wrote: post #24

Can anybody here help me understand why many teams are using MySQL as a KV store? (Uber did it recently, so assuming many others probably did it too, network effect) I personally love MySQL. Just want to understand what makes MySQL a great KV store as opposed to more seemingly specialized systems like Redis?

Less complexity, I imagine. If you've already got a beefy MySQL setup, then maintaining a separate Redis setup will introduce quite a bit of redundancy and chances for failure. Given that the big SQL databases have key value engines these days, you're going to get decent performance out of them as well.

Re: Moving persistent data out of Redis

#30

Post by sacheendra »

artursapek wrote: post #11
antirez wrote: post #3

Wow I'm learning today that Github used Redis for persistent data, now that they moved away :-) Anyway very happy that Redis helped to run such an important site. From the blog post it looks like that for certain things to move away from Redis was hard even if they are very skilled with MySQL, this is a good thing from the POV of Redis since it means that Redis allows to model certain things easily. However they want…

Pretty cool that redis was helping to host its own source code & development.

A lot of software help host their own source code and development. Just considering Github, 1. MySQL is hosted on Github. 2. I think they use elasticsearch and it is hosted on github. and lots of others Software is pretty cool like that!
Post reply on HN