Live data from Hacker News

Scaling the GitLab database

about.gitlab.com

21–30 of 114 posts

Re: Scaling the GitLab database

#21

Other options available: - CockroachDB 1.1 - AWS RDS Aurora PostgreSQL-compatibility [1] https://news.ycombinator.com/item?id=15458900 [2] https://news.ycombinator.com/item?id=13072861

CockroachDB is a really cool product, but I think by all accounts it is still nowhere near performant enough to be used in production.

Re: Scaling the GitLab database

#22
post #10
post #8

Earlier quoted context omitted.

User mentioned "trying". With an app that gets this much use, it might be reasonable to take a small piece and fork all writes to both the current + an experimental DB and then A/B test the reads. If this type of abstraction requires "a complete rewrite of GitLab" for "trying NewSQL DBs" then something is amiss.

Given that Gitlab is open source, adding a NewSQL DB into the mix would be pushing that dependency to downstream users, who would then have to take on the maintenance burden. It may end up being the right call from a performance standpoint but just too complex to ask users to manage.

[deleted]

Re: Scaling the GitLab database

#23
post #6

This article is also very useful in showing just how far you can push Postgres _without_ reaching for any of these optimizations. I've seen too many projects worry about these things very early on in their lifecycle, when in reality they are no where close to having enough traffic to cause a problem.

And almost certainly you can push postgresql even more by throwing hardware (ie. RAM) at the problem, although this is probably the point when it stops being worthwhile for reasonably sane applications (I've seen deployments where single pgsql server with 2TB+ RAM was cheaper solution than this kind of optimisation, due to all the technical debt in applications)

Re: Scaling the GitLab database

#25
post #6

This article is also very useful in showing just how far you can push Postgres _without_ reaching for any of these optimizations. I've seen too many projects worry about these things very early on in their lifecycle, when in reality they are no where close to having enough traffic to cause a problem.

Great point.

Is easy to overlook how much you can do with a decent RDBMS. And for the small data that gitlab use, I believe still exist a lot of big wins on performance.

Is just that the new generation not pay much attention to Sql databases...

P.D: I don't mean the gitlab developers, just on general

Re: Scaling the GitLab database

#26
post #17

Other options available: - CockroachDB 1.1 - AWS RDS Aurora PostgreSQL-compatibility [1] https://news.ycombinator.com/item?id=15458900 [2] https://news.ycombinator.com/item?id=13072861

Aurora RDS-PG still isn't GA, as in isn't available in all datacenters. Edit: Apparently it is GA as of 6 days ago, but still only available in 4 regions https://aws.amazon.com/blogs/aws/now-available-amazon-aurora...

There are only 9 regions with at least 3 AZs (as required by Aurora design), it's available in 4 of them.

Re: Scaling the GitLab database

#27
post #6

This article is also very useful in showing just how far you can push Postgres _without_ reaching for any of these optimizations. I've seen too many projects worry about these things very early on in their lifecycle, when in reality they are no where close to having enough traffic to cause a problem.

The whole point why you should worry from the beginning is so that you don't have to re-work everything and put a huge risk to the business when you have to do it.

Re: Scaling the GitLab database

#28
Apparently I should stop using gitlab.com By choosing to not shard your data you have basically put a cap on scalability... or at least one which will be reached faster than if you were to decide to shard.

Re: Scaling the GitLab database

#29
post #6

This article is also very useful in showing just how far you can push Postgres _without_ reaching for any of these optimizations. I've seen too many projects worry about these things very early on in their lifecycle, when in reality they are no where close to having enough traffic to cause a problem.

The whole point why you should worry from the beginning is so that you don't have to re-work everything and put a huge risk to the business when you have to do it.

Certainly. However, there's a difference between keeping scalability in mind, and actually implementing/maintaining a complex database scaling system at the onset of a project.

Edit: I would also add that most projects I've seen tend to undergo one or more large refactors / redesigns before growing to a size where complex DB scaling is needed. This is, of course, speaking from the perspective of a small startup or pet project. If you're writing a new feature for an existing product where you can assume that it'll have lots of users, then you would obviously build scaling right from the start.

Re: Scaling the GitLab database

#30
post #6

This article is also very useful in showing just how far you can push Postgres _without_ reaching for any of these optimizations. I've seen too many projects worry about these things very early on in their lifecycle, when in reality they are no where close to having enough traffic to cause a problem.

The whole point why you should worry from the beginning is so that you don't have to re-work everything and put a huge risk to the business when you have to do it.

Instead you appear to be advocating for over engineering a solution for a problem you don’t yet have where the opportunity costs are probably features customers can use. That is also a huge risk to a business, but one that is immediate.
Post reply on HN