Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling the GitLab database

about.gitlab.com

Post reply on HN
Print view View original

71–80 of 114 posts

Re: Scaling the GitLab database

#71

Post by awj »

YorickPeterse wrote: post #53
RandomBK wrote: post #6

This article is also very useful in showing just how far you can push Postgres _without_ reaching for any of these optimizations. I've seen too many projects worry about these things very early on in their lifecycle, when in reality they are no where close to having enough traffic to cause a problem.

Yes, you can typically push PostgreSQL very far while still using a fairly simple setup (e.g. no sharding). Unfortunately too many times people have this mindset that a slow application is the result of a slow/bad database (as in "it's an RDBMS and RDBMS' don't scale"), and not the result of it being misused (e.g. badly written queries, lack of proper indexes, that sort of thing). At GitLab it took quite a while to g…

1. Write web application with little to no idea how an RDBMS works

2. Run into inevitable performance problems

3. Decide that your database is at fault, rewrite in NoSQL

4. Buy into NoSQL hype, push 5x resources on your NoSQL solution

5. Profit?

There are NoSQL options that are great for lots of things. It takes some significant expertise to know if your thing is one of those things. Expertise you probably don't have if you don't even have a moderately deep knowledge of RDBMS-es.

Re: Scaling the GitLab database

#72

Post by GrayShade »

nisa wrote: post #69
GrayShade wrote: post #42

Earlier quoted context omitted.

I remember installing GitLab on a small NAS-type box. It was a while ago, but on each start-up it ran a nodejs tool to pre-compile some assets, I suppose. On that machine, it took 10-15 minutes to start. Afterwards, the unicorn workers kept getting killed because the default memory limit (128 MB) wasn't enough to process more than literally a couple of requests. It did work, but pages took 2-5 seconds to load, which…

So just use cgit on your 128mb box and don't complain about gitlab? It's like installing Windows 10 on a machine with 1gb memory, starting Photoshop and complaining about swapping and that it's running slow. Gitlab runs fine out of the box with 4gb memory and when screwing with some provided knobs it's fine with 2gb or less memory. It's in the docs, it's in every FAQ, every Stackoverflow answer... so why the hate? 64…

https://about.gitlab.com/2015/04/21/gitlab-on-raspberry-pi-2...

My box has 16 GB of memory. The 128 MB value was the default value configured in unicorn. If you're not familiar with that (I wasn't), it has a feature to restart the web server workers when they end up using too much memory. That's what unicorn does and apparently it's what people do in Ruby web apps.

I've also used GitLab on a (real) server. It still feels too slow for my taste (pages load in 1-2s). No, it's not unusable, but I won't call that "fine".

Re: Scaling the GitLab database

#73

Post by GrayShade »

kuschku wrote: post #66
GrayShade wrote: post #42

Earlier quoted context omitted.

I remember installing GitLab on a small NAS-type box. It was a while ago, but on each start-up it ran a nodejs tool to pre-compile some assets, I suppose. On that machine, it took 10-15 minutes to start. Afterwards, the unicorn workers kept getting killed because the default memory limit (128 MB) wasn't enough to process more than literally a couple of requests. It did work, but pages took 2-5 seconds to load, which…

> For anyone needing a small Git Web UI, I suggest https://github.com/go-gitea/gitea . But be aware, gitea (like gogs) does NOT cache SQL queries, so you are heavily limited by that. I can’t get it to serve > 200 pageloads per second on my system, while even a normal Grails projects manages to serve 3000 (with more complicated queries).

Thanks for the heads-up. In my case I'm the single user, so it doesn't matter, but I'm sure it could handle a lot more.

It used to cache something, either SQL queries or rendered pages, but it might have stopped in the meanwhile.

You made me curious, so I just tested it: 102 rps on a project page, 365 rps on a user profile, as reported by wrk2.

Re: Scaling the GitLab database

#74

Post by awj »

nickh9000 wrote: post #28

Apparently I should stop using gitlab.com By choosing to not shard your data you have basically put a cap on scalability... or at least one which will be reached faster than if you were to decide to shard.

Web service architectures are like shoes. You want what fits now, not what you hope will fit later.

Chances are very good that preemptive scaling will just result in unnecessary development overhead and an architecture/set of abstractions that still fail to handle the expected load since they weren't battle tested against it.

Re: Scaling the GitLab database

#75

Post by mschuster91 »

> A side effect of using transaction pooling is that you cannot use prepared statements, as the PREPARE and EXECUTE commands may end up running in different connections; producing errors as a result.

Uhhh... WHAT? I'll simply sit by and watch for SQL injection vulnerabilities resulting from this change. Either you have a very good SQL query writer engine with rock solid escaping or you will get pwned by this.

Re: Scaling the GitLab database

#76

Post by [deleted] »

sytse wrote: post #58
teraflop wrote: post #33

Earlier quoted context omitted.

That's probably the best "feature comparison" page I've ever seen. (At least, the best written by a creator of one of the products being compared.)

Thanks! We try to keep it fair. I love that our marketing team is on board with listing our missing features very publicly https://about.gitlab.com/features/#missing

[deleted]

Re: Scaling the GitLab database

#77

Post by dominotw »

sytse wrote: post #58
teraflop wrote: post #33

Earlier quoted context omitted.

That's probably the best "feature comparison" page I've ever seen. (At least, the best written by a creator of one of the products being compared.)

Thanks! We try to keep it fair. I love that our marketing team is on board with listing our missing features very publicly https://about.gitlab.com/features/#missing

> 2/3 of Enterprises Use GitLab

wow that a lot higher that i would have expected. Linked bitrise page used " randomly selected 10,000 apps as a base, that are getting built regularly on Bitrise" , not "enterprises".

How did you comeup with 2/3 number? First line of your about page is a lie?

Re: Scaling the GitLab database

#78

Post by YorickPeterse »

mschuster91 wrote: post #75

> A side effect of using transaction pooling is that you cannot use prepared statements, as the PREPARE and EXECUTE commands may end up running in different connections; producing errors as a result. Uhhh... WHAT? I'll simply sit by and watch for SQL injection vulnerabilities resulting from this change. Either you have a very good SQL query writer engine with rock solid escaping or you will get pwned by this.

You can have perfectly valid input escaping without needing prepared statements. Many ORMs handle this without the need for PREPARE/EXECUTE.

Re: Scaling the GitLab database

#79

Post by YorickPeterse »

awj wrote: post #71
YorickPeterse wrote: post #53

Earlier quoted context omitted.

Yes, you can typically push PostgreSQL very far while still using a fairly simple setup (e.g. no sharding). Unfortunately too many times people have this mindset that a slow application is the result of a slow/bad database (as in "it's an RDBMS and RDBMS' don't scale"), and not the result of it being misused (e.g. badly written queries, lack of proper indexes, that sort of thing). At GitLab it took quite a while to g…

1. Write web application with little to no idea how an RDBMS works 2. Run into inevitable performance problems 3. Decide that your database is at fault, rewrite in NoSQL 4. Buy into NoSQL hype, push 5x resources on your NoSQL solution 5. Profit? There are NoSQL options that are great for lots of things. It takes some significant expertise to know if your thing is one of those things. Expertise you probably don't have…

    > Buy into NoSQL hype, push 5x resources on your NoSQL solution
NoSQL is so last year, NewSQL is the future!

Re: Scaling the GitLab database

#80

Post by the_duke »

nisa wrote: post #69
GrayShade wrote: post #42

Earlier quoted context omitted.

I remember installing GitLab on a small NAS-type box. It was a while ago, but on each start-up it ran a nodejs tool to pre-compile some assets, I suppose. On that machine, it took 10-15 minutes to start. Afterwards, the unicorn workers kept getting killed because the default memory limit (128 MB) wasn't enough to process more than literally a couple of requests. It did work, but pages took 2-5 seconds to load, which…

So just use cgit on your 128mb box and don't complain about gitlab? It's like installing Windows 10 on a machine with 1gb memory, starting Photoshop and complaining about swapping and that it's running slow. Gitlab runs fine out of the box with 4gb memory and when screwing with some provided knobs it's fine with 2gb or less memory. It's in the docs, it's in every FAQ, every Stackoverflow answer... so why the hate? 64…

I'm running a Gitlab instance for a smallish dev team of 10.

I haven't gotten it to run reliably with any less than 8Gb, even though we don't have a whole lot of activity there.

It takes surprisingly long to start up too, even on a relatively beefy host.

I do like the product though, the all-in-one solution with code hosting, issues, code review (needs work though...) and CI is great. (Not using the deployment, orchestration stuff).

My biggest issue is that they seem to be too focused on pumping out new features.

There are 9k open issues just for CE. That's probably 12k+ for all products (runner, EE,...).

That either is a sign of very bad project management or a way to big agenda. Probably both.

Post reply on HN