Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving persistent data out of Redis

githubengineering.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Moving persistent data out of Redis

#51

Post by sjeanpierre »

rthrfrd wrote: post #10
antirez wrote: post #3

Wow I'm learning today that Github used Redis for persistent data, now that they moved away :-) Anyway very happy that Redis helped to run such an important site. From the blog post it looks like that for certain things to move away from Redis was hard even if they are very skilled with MySQL, this is a good thing from the POV of Redis since it means that Redis allows to model certain things easily. However they want…

FWIW we went through a very similar process to that documented here by Github (~3 months ago). It was entirely due to operational reasons and nothing to with shortcomings in Redis itself. MySQL was the master record for 99% of our data while Redis was the master record for the other 1% (as it happens it was also a kind of activity stream). Having the single 'master' reference for our data reduced complexity to a degr…

What method are you using to replicate from MySQL binlog to various other systems?

Re: Moving persistent data out of Redis

#52

Post by ec109685 »

adrr wrote: post #35

I don't know why people use redis as an LRU cache. Its a terrible LRU cache. Its eviction algorithm isn't true LRU and does sampling which may cause new keys to get incorrectly convicted. LRU is also really slow being single threaded.

Why does single threaded make it slow?

Also, 3.0 has a better LRU algorithm. https://redis.io/topics/lru-cache

Re: Moving persistent data out of Redis

#53

Post by vacri »

sacheendra wrote: post #30
artursapek wrote: post #11

Earlier quoted context omitted.

Pretty cool that redis was helping to host its own source code & development.

A lot of software help host their own source code and development. Just considering Github, 1. MySQL is hosted on Github. 2. I think they use elasticsearch and it is hosted on github. and lots of others Software is pretty cool like that!

I'm pretty sure that git's source is also version-controlled in git :)

Re: Moving persistent data out of Redis

#54

Post by vacri »

tschellenbach wrote: post #19

Wonder why they didn't use Cassandra for this use case.

Last year I was setting up a trial of Cassandra for something, going through the usual swearing of a new tool not quite working as expected (eg by default picking a random port for inter-node communication)... and the next desk over, a non-tech colleague called Cassandra kept hearing me mutter angrily about 'cassandra' and wondered what she'd done. Whoops :)

Re: Moving persistent data out of Redis

#55

Post by cookiecaper »

jhgg wrote: post #21

We've recently had to move away from redis for persistent data storage at work too - opting instead to write a service layer ontop of cassandra for storing data. Redis was tremendous in our journey up there - but one of the shortcomings is that it isn't as easy to scale-up as cassandra is if you haven't designed your system to scale-up on redis from when it was built (which we didn't) - instead of re-architecting for…

>instead of re-architecting for a redis-cluster setup, we decided to move the component to a clustered microservice written in go, that sits as a memory-cache & write buffer infront of cassandra for hot, highly mutated data.

Somehow setting up a Redis cluster and doing whatever you have to do to distribute/shard your keys effectively (which afaik is not much) does sound a little more efficient than rewriting a clustered microservice in Go with a Cassandra backend. Redis clustering is actually quite easy.

Forgive me if I seem grumpy. My recent experiences have caused the "We had a minor issue, so we redid everything in a Totally Cool Super-Neato New Stack That Integrates All The Hiring Manager's Favorite Buzzwords!" perspective to become a bit grating.

Redis is one of the few new pieces of infrastructure over the last 10 years that's truly deserving of its position.

Re: Moving persistent data out of Redis

#56

Post by lostcolony »

tschellenbach wrote: post #19

Wonder why they didn't use Cassandra for this use case.

They told you their motivations. Reduce complexity, and take advantage of their MySQL knowledge. Cassandra would fail both of those if they didn't have a lot of MySQL knowledge, since setting it up and maintaining it would increase complexity by introducing a tool they did not know.

Re: Moving persistent data out of Redis

#57

Post by eriknstr »

adrr wrote: post #49
zaius wrote: post #47

Earlier quoted context omitted.

What should people use?

Memcached is a LRU cache.

I went and searched for memcached redis because up until now I haven't gotten around to checking out the differences. Here are some things I found.

http://antirez.com/news/94

http://stackoverflow.com/questions/10558465/memcached-vs-red...

http://www.infoworld.com/article/3063161/application-develop...

Re: Moving persistent data out of Redis

#58

Post by jhgg »

cookiecaper wrote: post #55
jhgg wrote: post #21

We've recently had to move away from redis for persistent data storage at work too - opting instead to write a service layer ontop of cassandra for storing data. Redis was tremendous in our journey up there - but one of the shortcomings is that it isn't as easy to scale-up as cassandra is if you haven't designed your system to scale-up on redis from when it was built (which we didn't) - instead of re-architecting for…

>instead of re-architecting for a redis-cluster setup, we decided to move the component to a clustered microservice written in go, that sits as a memory-cache & write buffer infront of cassandra for hot, highly mutated data. Somehow setting up a Redis cluster and doing whatever you have to do to distribute/shard your keys effectively (which afaik is not much) does sound a little more efficient than rewriting a cluste…

My post above describes the main reason for moving from redis - the fact that data for inactive users doesn't need to be memory perpetually. :P

Re: Moving persistent data out of Redis

#59

Post by nvartolomei »

activatedgeek wrote: post #24

Can anybody here help me understand why many teams are using MySQL as a KV store? (Uber did it recently, so assuming many others probably did it too, network effect) I personally love MySQL. Just want to understand what makes MySQL a great KV store as opposed to more seemingly specialized systems like Redis?

Operational complexity at scale and prior experience with it. It is one thing to run one instance system and completely different story to run a clustered setup for large amounts of data with performance consideration.

MySQL also proved very good at scale, at Facebook, YouTube, Uber, etc. and there are a lot of people with experience running it.

Uber wrote a very good article about their switch, for them it was more about performance, though a little bit controversial.

Re: Moving persistent data out of Redis

#60

Post by toomuchtodo »

activatedgeek wrote: post #24

Can anybody here help me understand why many teams are using MySQL as a KV store? (Uber did it recently, so assuming many others probably did it too, network effect) I personally love MySQL. Just want to understand what makes MySQL a great KV store as opposed to more seemingly specialized systems like Redis?

> Just want to understand what makes MySQL a great KV store

Its not, it just happens to be good enough, which matters a lot for operational expertise/costs/etc.

For example, you can store hundreds of millions of KV rows in an InnoDB table and still have <1-3ms response times on queries, while having persistence built in. Perfect is the enemy of good enough.

Post reply on HN