Has anyone written about or noticed the performance differences between Gitlab and GitHub?
They're both Rails-based applications but I find page load times on Gitlab in general to be horrific compared to GitHub.
My notes on Gitlab's Postgres schema design (2022)
31–40 of 170 posts
Re: My notes on Gitlab's Postgres schema design (2022)
#32Is it just me that thinks in general schema design and development is stuck in the stone ages? I mainly know dotnet stuff, which does have migrations in EF (I note the point about gitlab not using this kind of thing because of database compatibility). It can point out common data loss while doing them. However, it still is always quite scary doing migrations, especially bigger ones refactoring something. Throw into t…
One thing I like about hand designing schema is it makes you sit down and make very clear choices about what your data is, how it interrelates, and how you’ll use it. You understand your own goals more clearly.
So many people I encounter seem to think it’s the code that’s important when building the back end of an application. You see this when people discussing database schemas start comparing, say, rails to hibernate. But ORMs emphasise code instead of data, which in my experience is a big mistake.
In my experience, getting the data structures right is 99% of the battle. If you get that right, the code that follows is simple and obvious.
For database applications, this means getting the schema right. To this end, I always start with the underlying table structures, and only start coding once I understand how the various tables are going to interact.
Sadly, too many people think of the database as the annoying hoops we jump through in order to store the results of our code. In my world, the code I write is the minimum required to safely manipulate the database; it’s the data that counts.
Some people seem to think I’m weird for starting with the database (and for using plpgsql), but I think it’s actually a superpower.
Re: My notes on Gitlab's Postgres schema design (2022)
#33Earlier quoted context omitted.
This largely depends on the disk. I wouldn't expect that to take 30mins on a modern NVME drive, but of course it depends on table size.
Disk wasn't the limit in my case, index creation is single threaded.
Which is still very IO bound... Wonder what kinda IOPS you were observing? Also they make pretty fast CPUs these days :)
Re: My notes on Gitlab's Postgres schema design (2022)
#34> For example, Github had 128 million public repositories in 2020. Even with 20 issues per repository it will cross the serial range. Also changing the type of the table is expensive. I expect the majority of those public repositories are forks of other repositories, and those forks only exist so someone could create pull requests against the main repository. As such, they won't ever have any issues, unless someone m…
Do we know for sure if gitlab cloud uses a multi-tenanted database, or a db per user/customer/org? In my experience products that offer both a self hosted and cloud product tend to prefer a database per customer, as this greatly simplifies the shared parts of the codebase, which can use the same queries regardless of the hosting type.
If they use a db per customer then no one will ever approach those usage limits and if they do they would be better suited to a self hosted solution.
Re: My notes on Gitlab's Postgres schema design (2022)
#35Is it just me that thinks in general schema design and development is stuck in the stone ages? I mainly know dotnet stuff, which does have migrations in EF (I note the point about gitlab not using this kind of thing because of database compatibility). It can point out common data loss while doing them. However, it still is always quite scary doing migrations, especially bigger ones refactoring something. Throw into t…
Not a silver bullet for every project but the Django ORM largely solves this with its migrations. You define your table classes and it just generates the migrations. Throw in a type checker and you're in pretty good shape. Rust also has sqlx which will type check your code against the DB.
An ORM is NEVER the solution, ever ever ever. Repeat after me: ORMs are not the solution to this problem. They work in your little toy apps with 4 customers but they are nothing but pain on real enterprise grade software.
Re: My notes on Gitlab's Postgres schema design (2022)
#36> 1 quintillion is equal to 1000000000 billions
it is pretty wild that we generally choose between int32 and int64. we really ought to have a 5 byte integer type which would support cardinalities of ~1T
Re: My notes on Gitlab's Postgres schema design (2022)
#37The point about the storage size of UUID columns is unconvincing. 128 bits vs. 64 bits doesn't matter much when the table has five other columns. A much more salient concern for me is performance. UUIDv4 is widely supported but is completely random, which is not ideal for index performance. UUIDv7[0] is closer to Snowflake[1] and has some temporal locality but is less widely implemented. There's an orthogonal approac…
Would it ever make sense to have a uuidv7 as primary key but then anther slug field for a public-id, e.g. one that is shorter and better in a url or even allowing user to customize it?
Re: My notes on Gitlab's Postgres schema design (2022)
#38Has anyone written about or noticed the performance differences between Gitlab and GitHub? They're both Rails-based applications but I find page load times on Gitlab in general to be horrific compared to GitHub.
I mean GitHub in general has been pretty reliable minus the two outages they had last year and is usually pretty performant or I wouldn’t use their keyboard shortcuts.
There are some complaints here from a former dev about gitlab that might provide insight into its culture and lack of regard for performance: https://news.ycombinator.com/item?id=39303323
Ps: I do not use gitlab enough to notice performance issues but thought you might appreciate the article
Re: My notes on Gitlab's Postgres schema design (2022)
#39[dead]
Re: My notes on Gitlab's Postgres schema design (2022)
#40> 1 quintillion is equal to 1000000000 billions it is pretty wild that we generally choose between int32 and int64. we really ought to have a 5 byte integer type which would support cardinalities of ~1T
Yeah it doesn't make sense to pick something that's not a power of 2 unless you are packing it.