Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving persistent data out of Redis

githubengineering.com

Post reply on HN
Print view View original

41–50 of 77 posts

Re: Moving persistent data out of Redis

#41

Post by why-el »

antirez wrote: post #3

Wow I'm learning today that Github used Redis for persistent data, now that they moved away :-) Anyway very happy that Redis helped to run such an important site. From the blog post it looks like that for certain things to move away from Redis was hard even if they are very skilled with MySQL, this is a good thing from the POV of Redis since it means that Redis allows to model certain things easily. However they want…

I think this is key:

"We needed something that would work for both github.com and GitHub Enterprise, so we decided to lean on our operational experience with MySQL."

Re: Moving persistent data out of Redis

#42

Post by sytse »

marcinkuzminski wrote: post #39
sytse wrote: post #38

Earlier quoted context omitted.

Thanks for asking. The issue you referred to talks about users on a single machine. Unlike GitHub you can run a cluster of application servers with GitLab https://about.gitlab.com/high-availability/ Some of our users have 25k+ users on their cluster. We know GitLab can scale to 100k users because we run GitLab Enterprise Edition without modifications on GitLab.com GitLab.com currently has much more than 100k users an…

can you share how many machines handles gitlab.com currently?

It is around 100 of machines. Commonly it is a couple of thousand active users per application server, but your mileage may vary based on many things.

Re: Moving persistent data out of Redis

#44

Post by john832 »

sytse wrote: post #38
john832 wrote: post #37

Earlier quoted context omitted.

Does GitLab really handle 100k users? Where is this indicated? https://gitlab.com/gitlab-org/gitlab-ce/issues/26405#note_20...

Thanks for asking. The issue you referred to talks about users on a single machine. Unlike GitHub you can run a cluster of application servers with GitLab https://about.gitlab.com/high-availability/ Some of our users have 25k+ users on their cluster. We know GitLab can scale to 100k users because we run GitLab Enterprise Edition without modifications on GitLab.com GitLab.com currently has much more than 100k users an…

> Unlike GitHub

GitHub seems to have a clustering product that came out a year ago, and it looks like IBM has reported they're running over 13,000 users on it (back in August, i imagine it's closer to 20,000 by now): https://www.ibm.com/blogs/bluemix/2016/08/ibm-internal-githu...

> We know GitLab can scale to 100k users because we run GitLab Enterprise Edition without modifications on GitLab.com

What does "modification" mean in this context? Beyond recommended specifications listed at https://docs.gitlab.com/ee/install/requirements.html#cpu and https://docs.gitlab.com/ee/install/requirements.html#memory? it doesnt list anything above 40,000 users. Beyond that, the HA documentation (https://docs.gitlab.com/ce/administration/high_availability/...) isn't _really_ active/active HA, it says it is but it's not true. True active/active would mean that you wouldn't rely on a shared NFS server, postgres, or redis server.

However, with https://help.github.com/enterprise/2.8/admin/guides/clusteri... it sounds like my organization can scale to well over 100,000 by adding more clustering nodes instead of trying to figure out independently how to scale services like redis or postgres or a shared NFS server.

> without much tuning.

I'd love to hear a comparison between the two products, and also what kind of tuning you've done on gitlab.com to support those user numbers. Would love to see that on the documentation to support the open source way!

Re: Moving persistent data out of Redis

#45

Post by rthrfrd »

parthdesai wrote: post #12
rthrfrd wrote: post #10

Earlier quoted context omitted.

FWIW we went through a very similar process to that documented here by Github (~3 months ago). It was entirely due to operational reasons and nothing to with shortcomings in Redis itself. MySQL was the master record for 99% of our data while Redis was the master record for the other 1% (as it happens it was also a kind of activity stream). Having the single 'master' reference for our data reduced complexity to a degr…

just asking for some info, but how do you make sure that multiple of your db systems are in sync (specifically interested in MySql and elasticsearch)? Hope it's alright to ask you that.

In the case of ES the short answer is; we don't. We have fault tolerance in our replication system to guarantee eventual consistency instead. I would say using ES as a consistent source of data isn't really playing to its strengths so we don't use it that way. The consistency you want is determined at read time: If you need consistency then hit MySQL, but for our use case that almost never happens as eventual consistency is usually instantaneous enough.

Our other tool is to decouple lookup (which objects to fetch) and population (what data to return for each object). You can mix and match, e.g. do a lookup against an inconsistent ES but still get consistent objects by populating from MySQL (or vice versa). As others have alluded to it depends entirely on the requirements for the result set.

Re: Moving persistent data out of Redis

#47

Post by zaius »

adrr wrote: post #35

I don't know why people use redis as an LRU cache. Its a terrible LRU cache. Its eviction algorithm isn't true LRU and does sampling which may cause new keys to get incorrectly convicted. LRU is also really slow being single threaded.

What should people use?

Re: Moving persistent data out of Redis

#48

Post by twblalock »

adrr wrote: post #35

I don't know why people use redis as an LRU cache. Its a terrible LRU cache. Its eviction algorithm isn't true LRU and does sampling which may cause new keys to get incorrectly convicted. LRU is also really slow being single threaded.

A lot of uses of Redis are suboptimal, but easy to set up and good enough. For example, "true" LRU is not always needed -- an approximation of LRU is sometimes sufficient. It's often not worth the time and money to do better.

When you need to scale past what Redis can provide, you can move on to a different solution, as Github has done.

Re: Moving persistent data out of Redis

#49

Post by adrr »

zaius wrote: post #47
adrr wrote: post #35

I don't know why people use redis as an LRU cache. Its a terrible LRU cache. Its eviction algorithm isn't true LRU and does sampling which may cause new keys to get incorrectly convicted. LRU is also really slow being single threaded.

What should people use?

Memcached is a LRU cache.

Re: Moving persistent data out of Redis

#50

Post by rhizome »

activatedgeek wrote: post #24

Can anybody here help me understand why many teams are using MySQL as a KV store? (Uber did it recently, so assuming many others probably did it too, network effect) I personally love MySQL. Just want to understand what makes MySQL a great KV store as opposed to more seemingly specialized systems like Redis?

For me a primary reason would be to avoid platform spread.
Post reply on HN