Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling the GitLab database

about.gitlab.com

Post reply on HN
Print view View original

11–20 of 114 posts

Re: Scaling the GitLab database

#11

Post by kodablah »

mnutt wrote: post #10
kodablah wrote: post #8

Earlier quoted context omitted.

User mentioned "trying". With an app that gets this much use, it might be reasonable to take a small piece and fork all writes to both the current + an experimental DB and then A/B test the reads. If this type of abstraction requires "a complete rewrite of GitLab" for "trying NewSQL DBs" then something is amiss.

Given that Gitlab is open source, adding a NewSQL DB into the mix would be pushing that dependency to downstream users, who would then have to take on the maintenance burden. It may end up being the right call from a performance standpoint but just too complex to ask users to manage.

This burden already exists with dependencies (and we should probably say downstream "admins" instead of "users" to clarify). A self-hosted app should not be hamstrung by its initial dependency choices. If tested and implemented properly it can go from optional experimental flag, to multiple-storage-backends-supported abstraction, to [maybe] even we're-migrating-to-new-storage-backend, etc. The primary downside in these cases is the flexibility often provided to plugin and extension writers which marries them to early tech choices. The more admins that demand low change count on forward movement are indirectly the reason many companies choose to offer SaaS only where the visibility and flexibility are removed.

Note, I'm not saying this is the case with GitLab at all and I assume Postgres will remain the primary choice for most of their DB uses in perpetuity, but for some use cases there is better tech to use and the only reason to say no shouldn't just be a dependency addition (though it is often a good reason among many to say no).

Re: Scaling the GitLab database

#12

Post by craigkerstiens »

twojustin wrote: post #9

My team is currently using the hosted cituscloud, which uses PG Bouncer. They note that the reason for sharding is because of high writes. We've actually seen big benefits for moving over to a sharded setup via Citus just as much for the read performance. By sharding by customer we're able to more effectively leverage Postgres caching and elastically scale. Since switching over, our database has performed and scaled…

Craig from Citus here. Thanks for the kind words Justin, as you mention there is some upfront work, but once it's in place it becomes fairly manageable.

We've been working to make that upfront work easier as well with libraries that allow things to be more drop-in (ActiveRecord-multi-tenant: https://github.com/citusdata/activerecord-multi-tenant and Django-multitenant: https://github.com/citusdata/django-multitenant)

Re: Scaling the GitLab database

#13

Post by derefr »

tschellenbach wrote: post #3

Running PG bouncer is a very basic optimization. You typically start out with PG bouncer in your stack if you have experience running Postgres. If you're new to running your own postgres databases you should also check out Wall-e: https://github.com/wal-e/wal-e And the awesome pg stat statements https://www.postgresql.org/docs/10/static/pgstatstatements.h...

I’ve set up both pgbouncer and wal-e in the past, and they’re both worth it to have from effectively the very beginning.

But it leaves me wondering: is there any sort of packaging of Postgres and these “friends” into a single opinionated virtual appliance, such that I could just stick up a couple of VMs with the same image with different role tags, and get a good cluster with automatic transparent N:M proxying, automatic transparent backup-and-restore, etc?

In other words, is there a product that is to Postgres as Gitlab CE is to Github (or like Dokku is to Heroku): a “host it yourself, but easily, and starting from a scale of 1 with no overhead” equivalent of a SaaS service?

Re: Scaling the GitLab database

#15

Post by snuxoll »

derefr wrote: post #13
tschellenbach wrote: post #3

Running PG bouncer is a very basic optimization. You typically start out with PG bouncer in your stack if you have experience running Postgres. If you're new to running your own postgres databases you should also check out Wall-e: https://github.com/wal-e/wal-e And the awesome pg stat statements https://www.postgresql.org/docs/10/static/pgstatstatements.h...

I’ve set up both pgbouncer and wal-e in the past, and they’re both worth it to have from effectively the very beginning. But it leaves me wondering: is there any sort of packaging of Postgres and these “friends” into a single opinionated virtual appliance, such that I could just stick up a couple of VMs with the same image with different role tags, and get a good cluster with automatic transparent N:M proxying, autom…

If you're deploying on K8S at least Crunchy Data has you covered [0], I haven't used it myself but it's the closest I've been able to find without going with something like EnterpriseDB.

I'm pretty sure there's some ansible/puppet/chef/saltstack code out there to build something similar as well.

[0] https://github.com/CrunchyData/crunchy-containers

Re: Scaling the GitLab database

#16

Post by sb8244 »

twojustin wrote: post #9

My team is currently using the hosted cituscloud, which uses PG Bouncer. They note that the reason for sharding is because of high writes. We've actually seen big benefits for moving over to a sharded setup via Citus just as much for the read performance. By sharding by customer we're able to more effectively leverage Postgres caching and elastically scale. Since switching over, our database has performed and scaled…

One of the things that I'm now excited about for citus is multi node IO capacity. It should improve write and read latencies in the situation where a single node disk is getting saturated.

* Edit. I AM excited for this. Typo'd now = not previously

Re: Scaling the GitLab database

#17

Post by web007 »

karmakaze wrote: post #14

Other options available: - CockroachDB 1.1 - AWS RDS Aurora PostgreSQL-compatibility [1] https://news.ycombinator.com/item?id=15458900 [2] https://news.ycombinator.com/item?id=13072861

Aurora RDS-PG still isn't GA, as in isn't available in all datacenters.

Edit: Apparently it is GA as of 6 days ago, but still only available in 4 regions https://aws.amazon.com/blogs/aws/now-available-amazon-aurora...

Re: Scaling the GitLab database

#18

Post by craigkerstiens »

sb8244 wrote: post #16
twojustin wrote: post #9

My team is currently using the hosted cituscloud, which uses PG Bouncer. They note that the reason for sharding is because of high writes. We've actually seen big benefits for moving over to a sharded setup via Citus just as much for the read performance. By sharding by customer we're able to more effectively leverage Postgres caching and elastically scale. Since switching over, our database has performed and scaled…

One of the things that I'm now excited about for citus is multi node IO capacity. It should improve write and read latencies in the situation where a single node disk is getting saturated. * Edit. I AM excited for this. Typo'd now = not previously

Not sure I fully follow on the multi node IO capacity, can you share a bit more on the workload that you're concerned about?

Edit: Thanks for the clarification, the typo part in particular through me off, makes sense now.

Re: Scaling the GitLab database

#19

Post by craigkerstiens »

tschellenbach wrote: post #3

Running PG bouncer is a very basic optimization. You typically start out with PG bouncer in your stack if you have experience running Postgres. If you're new to running your own postgres databases you should also check out Wall-e: https://github.com/wal-e/wal-e And the awesome pg stat statements https://www.postgresql.org/docs/10/static/pgstatstatements.h...

If you are already familiar with wal-e, or even if not, you might want to consider taking a look at wal-g[1]. Wal-g is a newer edition of wal-e written in go that we've seen can have up to 7x performance improvements[2].

[1] https://github.com/wal-g/wal-g

[2] https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-...

Re: Scaling the GitLab database

#20

Post by sb8244 »

craigkerstiens wrote: post #18
sb8244 wrote: post #16

Earlier quoted context omitted.

One of the things that I'm now excited about for citus is multi node IO capacity. It should improve write and read latencies in the situation where a single node disk is getting saturated. * Edit. I AM excited for this. Typo'd now = not previously

Not sure I fully follow on the multi node IO capacity, can you share a bit more on the workload that you're concerned about? Edit: Thanks for the clarification, the typo part in particular through me off, makes sense now.

Let's say that you're using RDS and have a single box capable of ~25k IOPS. If you have 16 boxes capable of some number of IOPS (let's say 15k), then your total system capacity is significantly higher than 25k. On read workloads where pages need pulled from disk (high read IOPS), this should see improvement in general.

Secondaries cover this case for Gitlab, it seems, but that comes with a set of caveats as well (namely async availability of data).

Post reply on HN