Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling the GitLab database

about.gitlab.com

Post reply on HN
Print view View original

81–90 of 114 posts

Re: Scaling the GitLab database

#81

Post by mschuster91 »

YorickPeterse wrote: post #78
mschuster91 wrote: post #75

> A side effect of using transaction pooling is that you cannot use prepared statements, as the PREPARE and EXECUTE commands may end up running in different connections; producing errors as a result. Uhhh... WHAT? I'll simply sit by and watch for SQL injection vulnerabilities resulting from this change. Either you have a very good SQL query writer engine with rock solid escaping or you will get pwned by this.

You can have perfectly valid input escaping without needing prepared statements. Many ORMs handle this without the need for PREPARE/EXECUTE.

> You can have perfectly valid input escaping without needing prepared statements.

People have bypassed this far too often for my taste, and there is a reason why SQL (and other injections) are Top 1 of OWASP (1st place actually, both in RC1 and 2 of 2017, but also in the 2013 edition).

Never, ever trust any form of escaping or it will bite you. Hard.

Re: Scaling the GitLab database

#82

Post by jadbox »

twojustin wrote: post #9

My team is currently using the hosted cituscloud, which uses PG Bouncer. They note that the reason for sharding is because of high writes. We've actually seen big benefits for moving over to a sharded setup via Citus just as much for the read performance. By sharding by customer we're able to more effectively leverage Postgres caching and elastically scale. Since switching over, our database has performed and scaled…

For a customer DB, why was Citus required for your use-case? Do you have millions of users?

Re: Scaling the GitLab database

#83

Post by elvinyung »

forgot-my-pw wrote: post #44
elvinyung wrote: post #21

Earlier quoted context omitted.

CockroachDB is a really cool product, but I think by all accounts it is still nowhere near performant enough to be used in production.

Alternative is Google Spanner. Also still new, but already in GA for 5 months now.

Well, at least Spanner has already been in development for 10 years.

Re: Scaling the GitLab database

#84

Post by ris »

nickh9000 wrote: post #27
RandomBK wrote: post #6

This article is also very useful in showing just how far you can push Postgres _without_ reaching for any of these optimizations. I've seen too many projects worry about these things very early on in their lifecycle, when in reality they are no where close to having enough traffic to cause a problem.

The whole point why you should worry from the beginning is so that you don't have to re-work everything and put a huge risk to the business when you have to do it.

No. No. Never do this.

You never actually know where your bottlenecks are going to be until they arrive and by designing everything with a "super scalable" architecture you will be making development ten times as painful and expensive as it needs to be while throwing away nice things that come "for free" and "just work" at the mid-low end like transactions.

Amazon and Netflix don't want to have to use their hideously complex and inefficient service architectures - they're forced to because of their scale.

Most people who engineer their systems for hyper-scale from the get-go never see a whiff of anything that looks remotely like high traffic. Often they go out of business before they get anywhere near that.

And once you get to serious scale, you really shouldn't still be running the code from back when you didn't really know what your business/product was.

Re: Scaling the GitLab database

#85

Post by ris »

mschuster91 wrote: post #81
YorickPeterse wrote: post #78

Earlier quoted context omitted.

You can have perfectly valid input escaping without needing prepared statements. Many ORMs handle this without the need for PREPARE/EXECUTE.

> You can have perfectly valid input escaping without needing prepared statements. People have bypassed this far too often for my taste, and there is a reason why SQL (and other injections) are Top 1 of OWASP (1st place actually, both in RC1 and 2 of 2017, but also in the 2013 edition). Never, ever trust any form of escaping or it will bite you. Hard.

> Never, ever trust any form of escaping or it will bite you. Hard.

This is an absurd dogmatic statement. Everybody trusts a multitude of escapings everywhere every day all over the web without even knowing it.

Re: Scaling the GitLab database

#86

Post by flukus »

mschuster91 wrote: post #75

> A side effect of using transaction pooling is that you cannot use prepared statements, as the PREPARE and EXECUTE commands may end up running in different connections; producing errors as a result. Uhhh... WHAT? I'll simply sit by and watch for SQL injection vulnerabilities resulting from this change. Either you have a very good SQL query writer engine with rock solid escaping or you will get pwned by this.

What is transaction pooling anyway? A transaction exists for a specific action, it shouldn't be shared among users or multiple connections. It sounds like they've taken transactions and removed the isolation.

I don't know what the data access is like, but I get the impression that it's like some "enterprise" patterns I've seen where data access is too "encapsulated", ie, each DAL opens it's own connection, pulls it's chunk of data and each request involves many of these instead of having each request open the connection and pass it down to the data access layers.

Re: Scaling the GitLab database

#87

Post by philliphaydon »

YorickPeterse wrote: post #53
RandomBK wrote: post #6

This article is also very useful in showing just how far you can push Postgres _without_ reaching for any of these optimizations. I've seen too many projects worry about these things very early on in their lifecycle, when in reality they are no where close to having enough traffic to cause a problem.

Yes, you can typically push PostgreSQL very far while still using a fairly simple setup (e.g. no sharding). Unfortunately too many times people have this mindset that a slow application is the result of a slow/bad database (as in "it's an RDBMS and RDBMS' don't scale"), and not the result of it being misused (e.g. badly written queries, lack of proper indexes, that sort of thing). At GitLab it took quite a while to g…

Lack of indexes vs way too many indexes that writes suffer.

Re: Scaling the GitLab database

#88

Post by dantiberian »

> Sharding would also affect the process of contributing changes to GitLab as every contributor would now have to make sure a shard key is present in their queries.

Why is this a problem? If you need to make a change to improve scalability, it doesn't seem unreasonable to ask contributors to follow code and performance guidelines.

Re: Scaling the GitLab database

#89

Post by maktouch »

We're moving away from gitlab because the performance is too much to bear. We're hearing "aaaaaarg I really hate gitlab" a few times a day.

The pipeline pages takes 7 seconds to load. It's really bad. We stopped using issues because just listing them was a chore. Pushing 1 file takes at least 30 seconds to 1 minute.

Anyways, the work falls on me to replace it. We're going with Phabricator and Jenkins.

Re: Scaling the GitLab database

#90

Post by josegonzalez »

maktouch wrote: post #89

We're moving away from gitlab because the performance is too much to bear. We're hearing "aaaaaarg I really hate gitlab" a few times a day. The pipeline pages takes 7 seconds to load. It's really bad. We stopped using issues because just listing them was a chore. Pushing 1 file takes at least 30 seconds to 1 minute. Anyways, the work falls on me to replace it. We're going with Phabricator and Jenkins.

Is that self-hosted Gitlab or their hosted version? Would you mind sharing details on your setup? Just curious as we're currently building out our own setup.
Post reply on HN