[deleted]
Scaling the GitLab database
61–70 of 114 posts
Re: Scaling the GitLab database
#62Earlier quoted context omitted.
I’ve set up both pgbouncer and wal-e in the past, and they’re both worth it to have from effectively the very beginning. But it leaves me wondering: is there any sort of packaging of Postgres and these “friends” into a single opinionated virtual appliance, such that I could just stick up a couple of VMs with the same image with different role tags, and get a good cluster with automatic transparent N:M proxying, autom…
If you're deploying on K8S at least Crunchy Data has you covered [0], I haven't used it myself but it's the closest I've been able to find without going with something like EnterpriseDB. I'm pretty sure there's some ansible/puppet/chef/saltstack code out there to build something similar as well. [0] https://github.com/CrunchyData/crunchy-containers
There's a postgres operator by Zalando which sets up postgres with WAL-E and auto failover using Patroni.
Re: Scaling the GitLab database
#63It's interesting to see what parts of the product GitLab is struggling with compared to GitHub, e.g. https://githubengineering.com/stretching-spokes/
I'm very impressed with the level GitHub database testing is at https://githubengineering.com/mysql-testing-automation-at-gi... We still got a lot of work ahead of us. But Yorick and Greg their work has been outstanding.
Regarding Spokes we're not planning something like that at this time. From https://gitlab.com/gitlab-org/gitaly/issues/650 "GitHub eventually moved to Spokes https://githubengineering.com/building-resilience-in-spokes/ that had multiple fileservers for the same repository. We will probably not need that. We run networked storage in the cloud where the public cloud provider is responsible for the redundancy of the files. To do cross availability zone failover we'll use GitLab Geo."
Re: Scaling the GitLab database
#64Not seeing any mention of using something like Memcached (or anything similar) in front of PG for caching. Is that because it's already been done and not in scope? Asking because for (at least) web applications, using Memcached (or similar) can significantly reduce the load on the backend database. :)
We have a lot of caching in Redis, but that connects with the Rails app, not in front of PG. This way it can store the result (for example a page fragment) instead of just the db query that is part of it.
Cool, just making sure. :)
Re: Scaling the GitLab database
#65Looking through the public Grafana dashboards I noticed that the database backups page is not showing any data points ( http://monitor.gitlab.net/dashboard/db/backups ). Is this meant to a public page?
There was an issue with the dashboard causing it to try and load data from the wrong place, this has been fixed. See https://gitlab.com/gitlab-com/infrastructure/issues/3115 for more info.
Re: Scaling the GitLab database
#66Earlier quoted context omitted.
> and the glaring problem of "ruby doesnt scale well" is everpresent GitLab has many performance problems (and many have been solved over the years), but Ruby has thus far not been one of them.
I remember installing GitLab on a small NAS-type box. It was a while ago, but on each start-up it ran a nodejs tool to pre-compile some assets, I suppose. On that machine, it took 10-15 minutes to start. Afterwards, the unicorn workers kept getting killed because the default memory limit (128 MB) wasn't enough to process more than literally a couple of requests. It did work, but pages took 2-5 seconds to load, which…
> For anyone needing a small Git Web UI, I suggest https://github.com/go-gitea/gitea.
But be aware, gitea (like gogs) does NOT cache SQL queries, so you are heavily limited by that. I can’t get it to serve > 200 pageloads per second on my system, while even a normal Grails projects manages to serve 3000 (with more complicated queries).
Re: Scaling the GitLab database
#67It's interesting to see what parts of the product GitLab is struggling with compared to GitHub, e.g. https://githubengineering.com/stretching-spokes/
I'm very impressed with the level GitHub database testing is at https://githubengineering.com/mysql-testing-automation-at-gi... We still got a lot of work ahead of us. But Yorick and Greg their work has been outstanding. Regarding Spokes we're not planning something like that at this time. From https://gitlab.com/gitlab-org/gitaly/issues/650 "GitHub eventually moved to Spokes https://githubengineering.com/building-re…
Thanks for chiming in, sytse and YorickPeterse!
Re: misuse, I remember this incident https://news.ycombinator.com/item?id=11245652 made me chuckle and cry a little. If you build it, it will be misused.
Re: Scaling the GitLab database
#68Funny thing is that they publicly own up to their performance problems in an unusual way: in their comparison with GitHub ( https://about.gitlab.com/comparison/ ) they list "Fast page load" as a feature that GitLab lacks and GitHub has. Nevertheless, the slowness is really annoying, especially because their product is so good on all other accounts. If scaling their database can help speed things up, I bet they will b…
The good news is that page loads are much better now. Our initial page load ping http://stats.pingdom.com/81vpf8jyr1h9/1902794 is better than GitHub.com http://stats.pingdom.com/81vpf8jyr1h9/1902795 We got work to do in the 99% and merge request page load but the overall situation has improve dramatically. We still got work to do in availability, so I changed the 'feature' to reflect this https://gitlab.com/gitlab-co…
Another GitLabber here.
We just kicked off a major effort to address the availability issue Sid mentions above. The highlights are that we're moving to GCP which should provide better underlying reliability. But interestingly we found that only about ~20% of our downtime minutes where from underlying infrastructure. Whereas ~70% came from features that didn't scale.
So the more exciting part of the project is to tighten the feedback loop between development and deployment with a continuous delivery pipeline. This may be obvious to some people, but it's harder to pull off when you've got an open source project, an on-prem product, and a large-scale SaaS sharing the same code base. I'm calling it "Open-core SaaS" and there are only a handful of companies that run a large, multi-tenant service based on an open source project.
Re: Scaling the GitLab database
#69Earlier quoted context omitted.
> and the glaring problem of "ruby doesnt scale well" is everpresent GitLab has many performance problems (and many have been solved over the years), but Ruby has thus far not been one of them.
I remember installing GitLab on a small NAS-type box. It was a while ago, but on each start-up it ran a nodejs tool to pre-compile some assets, I suppose. On that machine, it took 10-15 minutes to start. Afterwards, the unicorn workers kept getting killed because the default memory limit (128 MB) wasn't enough to process more than literally a couple of requests. It did work, but pages took 2-5 seconds to load, which…
So just use cgit on your 128mb box and don't complain about gitlab? It's like installing Windows 10 on a machine with 1gb memory, starting Photoshop and complaining about swapping and that it's running slow.
Gitlab runs fine out of the box with 4gb memory and when screwing with some provided knobs it's fine with 2gb or less memory.
It's in the docs, it's in every FAQ, every Stackoverflow answer... so why the hate?
64mb memory? Use gitolite. 256mb memory? Use cgit + gitolite 1gb memory? use gitea or whatever else there is. 4gb memory? use gitlab if you want.
I'm not affiliated but I can't stand the "gitlab doesn't run on my home wifi router - it sucks" posts here... it works pretty well if you read & acknowledge the documentation and requirements.
Re: Scaling the GitLab database
#70Earlier quoted context omitted.
> and the glaring problem of "ruby doesnt scale well" is everpresent GitLab has many performance problems (and many have been solved over the years), but Ruby has thus far not been one of them.
I remember installing GitLab on a small NAS-type box. It was a while ago, but on each start-up it ran a nodejs tool to pre-compile some assets, I suppose. On that machine, it took 10-15 minutes to start. Afterwards, the unicorn workers kept getting killed because the default memory limit (128 MB) wasn't enough to process more than literally a couple of requests. It did work, but pages took 2-5 seconds to load, which…
> I'm sure part of this is the fault of the language.
"Fault" is a pretty loaded word here. Pretty sure my old TI-86 also would struggle to run GitLab...