Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we decreased GitLab repo backup times from 48 hours to 41 minutes

about.gitlab.com

Post reply on HN
Print view View original

61–70 of 273 posts

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#61

Post by [deleted] »

nightfly wrote: post #52
bombela wrote: post #27

Earlier quoted context omitted.

> Be aware that even with these recommendations, syncing in this way has some risk since it bypasses Git’s normal integrity checking for repositories, so having backups is advised. You may also wish to do a git fsck to verify the integrity of your data on the destination system after syncing. https://git-scm.com/docs/gitfaq#_transfers It doesn't tell you how to make a backup safely though. On a personal scale, Syncth…

Syncthing is the only way I've ever corrupted a git repo before

[deleted]

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#62

Post by csnweb »

hiddew wrote: post #6

"fixed it with an algorithmic change, reducing backup times exponentially" If the backup times were O(n^2), are they now O(n^2 / 2^n)? I would guess not.

If you replace an n^2 algorithm with a log(n) lookup you get an exponential speed up. Although a hashmap lookup is usually O(1), which is even faster.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#63

Post by mdaniel »

tomxor wrote: post #16

One of the many reasons I moved to self hosting. I use ZFS to backup every 15 minutes, ... could do it even more frequently but that seems a little pointless. Also moved away from Gitlab because it's so damn slow.

It for sure has an excruciatingly slow UI, but I haven't yet been able to stomach the Ruby magick enough to dig in and find out if it's all of GitLab that's slow, and thus their culture of who-cares just propagates to the UI, or it's literally just the web frontend that's molasses yet because so many folks interact with it over the web that's where the conclusion ends up

I know their backend git proxy is written in golang, their runner agent is written in golang, it spawns CI jobs using containerd, written in golang, and they use postgresql and a redis-esque KV -- although for that part I do believe they're still using Sidekick (ruby) for doing job dispatch, so that one could very easily lolol back into not actioning tasks efficiently

GitHub Actions sure does enjoy just sitting there twiddling its thumbs when I push "run job," and is also both Ruby and their own crazypants ajax-y web framework, so I don't think it's exactly the shining star of performance itself

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#64

Post by hobofan »

nightfly wrote: post #52
bombela wrote: post #27

Earlier quoted context omitted.

> Be aware that even with these recommendations, syncing in this way has some risk since it bypasses Git’s normal integrity checking for repositories, so having backups is advised. You may also wish to do a git fsck to verify the integrity of your data on the destination system after syncing. https://git-scm.com/docs/gitfaq#_transfers It doesn't tell you how to make a backup safely though. On a personal scale, Syncth…

Syncthing is the only way I've ever corrupted a git repo before

I think that's why they specified the "BTRFS snapshots" part. Yes, directly syncing a .git directory seems like a recipe for disaster with how often I've seen individual files lagging to sync, but I guess with BTRFS snaphots one can ensure that only a consistent view of a git directory is being backed up and synced.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#65

Post by mdaniel »

treesknees wrote: post #17

Duplicate of https://news.ycombinator.com/item?id=44197061

Since that one only has one comment, I'm pretty sure that means the HN front-page is luck of the draw and I wouldn't have bothered even commenting about the other one

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#66

Post by bob1029 »

nonameiguess wrote: post #60
bob1029 wrote: post #50

I'm confused why you wouldn't simply snapshot the block-level device if the protocol of the information on top is going to cause this much headache. Quiescing git operations for block level activity is probably not trivial, but it sounds like an easier problem to solve to me. This is the approach I've taken with SQLite in production environments. Turn on WAL and the problem gets even easier to solve. Customer configu…

Gitlab isn't just a managed service. They release the software for self-hosted instances as well. There is no guarantee or requirement that users all run Gitlab on the same filesystem or even a filesystem that supports block level snapshots at all. Presumably, they want a universal backup system that works for all Gitlabs.

I've never heard of a .git folder that spanned multiple filesystems. It sounds like we are now conflating the git workspace with everything else in the product.

There are system requirements that a customer would be expected to adhere to if they wanted a valid enterprise support contract with one of these vendors.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#67

Post by EdwardCoffin »

LgWoodenBadger wrote: post #18

IME, it has always turned out to be the correct decision to eliminate any n^2 operation in anything I’ve written. I don’t write exotic algorithms, but it’s always astounding how small n needs to be to become observably problematic.

Bruce Dawson says: I like to call this Dawson’s first law of computing: O(n^2) is the sweet spot of badly scaling algorithms: fast enough to make it into production, but slow enough to make things fall down once it gets there.

https://bsky.app/profile/randomascii.bsky.social/post/3lk4c6...

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#68

Post by blharr »

cvoss wrote: post #47
School-Cotton wrote: post #8

Earlier quoted context omitted.

This is not the precise mathematical definition of exponential, but rather the colloquial one, where it just means "a lot".

You shouldn't use a word that can carry a precise mathematical meaning in a sentence that literally uses mathematical notation in order to speak precisely and then expect readers not to interpret the word in the precise mathematical way.

I somewhat agree, but for lack of a better word, what would you use? Quadratically doesn't have the same punch

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#70

Post by IshKebab »

cvoss wrote: post #47
School-Cotton wrote: post #8

Earlier quoted context omitted.

This is not the precise mathematical definition of exponential, but rather the colloquial one, where it just means "a lot".

You shouldn't use a word that can carry a precise mathematical meaning in a sentence that literally uses mathematical notation in order to speak precisely and then expect readers not to interpret the word in the precise mathematical way.

You should if you expect your readers to be normal humans who understand obvious context, and not pedantic HN readers who understand obvious context but delight in nit-picking it anyway.
Post reply on HN