Scaling the GitLab database • Hacker News Board

Re: Scaling the GitLab database

#111

Post by jjirsa » Wed, Nov 01, 2017, 3:27 AM UTC

elvinyung wrote: post #21

karmakaze wrote: post #14

Other options available: - CockroachDB 1.1 - AWS RDS Aurora PostgreSQL-compatibility [1] https://news.ycombinator.com/item?id=15458900 [2] https://news.ycombinator.com/item?id=13072861

CockroachDB is a really cool product, but I think by all accounts it is still nowhere near performant enough to be used in production.

Or, you know, use something like Cassandra that’s used at massive scales already (by Apple, Netflix, instagram, uber, Walmart, IBM, etc), and scales linearly to make future growth trivial.

Bonus points that rm -rf on a Cassandra server isn’t a problem, which seems like it could have been useful for gitlab.

Re: Scaling the GitLab database

#112

Post by the_duke » Wed, Nov 01, 2017, 7:31 AM UTC

sytse wrote: post #96

the_duke wrote: post #80

Earlier quoted context omitted.

I'm running a Gitlab instance for a smallish dev team of 10. I haven't gotten it to run reliably with any less than 8Gb, even though we don't have a whole lot of activity there. It takes surprisingly long to start up too, even on a relatively beefy host. I do like the product though, the all-in-one solution with code hosting, issues, code review (needs work though...) and CI is great. (Not using the deployment, orche…

I'm surprised to hear GitLab isn't running reliably with 8GB. Are you running the Omnibus installations? The open issues for CE https://gitlab.com/gitlab-org/gitlab-ce/issues do include feature requests.

I'm using the official docker image.

It's been very pain-less in setup and maintainance. Upgrades were mostly painless as well until now.

With 8GB for the docker container it's solid and pretty stable, but any less and behaviour would be very unstable (timeouts, very long request load times, ...).

Of course those 8GB are for the whole stack (Postgres, Redis, Nginx, Unicorn instances, ...), so I'll admit that 8GB for the Docker container is quite important context here. ;)

Re: Scaling the GitLab database

#113

Post by sytse » Wed, Nov 01, 2017, 3:52 PM UTC

the_duke wrote: post #112

sytse wrote: post #96

Earlier quoted context omitted.

I'm surprised to hear GitLab isn't running reliably with 8GB. Are you running the Omnibus installations? The open issues for CE https://gitlab.com/gitlab-org/gitlab-ce/issues do include feature requests.

I'm using the official docker image. It's been very pain-less in setup and maintainance. Upgrades were mostly painless as well until now. With 8GB for the docker container it's solid and pretty stable, but any less and behaviour would be very unstable (timeouts, very long request load times, ...). Of course those 8GB are for the whole stack (Postgres, Redis, Nginx, Unicorn instances, ...), so I'll admit that 8GB for…

Thanks for the answer. I would have thought that 4GB is enough. Thanks for providing a data point that it might not be.

Re: Scaling the GitLab database

#114

Post by nickh9000 » Thu, Nov 02, 2017, 4:14 PM UTC

jsmeaton wrote: post #30

nickh9000 wrote: post #27

Earlier quoted context omitted.

The whole point why you should worry from the beginning is so that you don't have to re-work everything and put a huge risk to the business when you have to do it.

Instead you appear to be advocating for over engineering a solution for a problem you don’t yet have where the opportunity costs are probably features customers can use. That is also a huge risk to a business, but one that is immediate .

Thinking about how you would shard your data is hardly over engineering. On some storage systems you are forced to anyway: Cassandra, dynamodb.

And the cost of building the solution from the onset is hardly a killer. The pain usually comes from operational complexity.