Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we decreased GitLab repo backup times from 48 hours to 41 minutes

about.gitlab.com

Post reply on HN
Print view View original

71–80 of 273 posts

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#71

Post by IshKebab »

pjmlp wrote: post #39

A very good example that writing code in C doesn't help for performance, when the algorithms or data structures aren't properly taken into consideration.

I would say C makes this sort of thing far more likely because it's usually a ton of effort to obtain suitable containers. In C++ or Rust they have plenty of things like `unordered_set`/`HashSet` built in, so people are much more likely to use it and not go "eh, I'll use a for loop".

In this case Git already had a string set, but it's still not standard so there's a good chance the original author just didn't know about it.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#72

Post by herpderperator »

rubit_xxx21 wrote: post #69

[flagged]

If the requirement is to check uniqueness, what assumptions could possibly cause a bug? In this case, why does it matter if the uniqueness is tested with a nested for loop or with a map? There are many identical ways to check uniqueness, some being faster than others.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#73

Post by amtamt »

bob1029 wrote: post #50

I'm confused why you wouldn't simply snapshot the block-level device if the protocol of the information on top is going to cause this much headache. Quiescing git operations for block level activity is probably not trivial, but it sounds like an easier problem to solve to me. This is the approach I've taken with SQLite in production environments. Turn on WAL and the problem gets even easier to solve. Customer configu…

> This is the approach I've taken with SQLite in production environments. Turn on WAL and the problem gets even easier to solve.

A few months back a better solution was provided by SQLite: sqlite3_rsync

https://www.sqlite.org/rsync.html

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#74

Post by bombela »

hobofan wrote: post #64
nightfly wrote: post #52

Earlier quoted context omitted.

Syncthing is the only way I've ever corrupted a git repo before

I think that's why they specified the "BTRFS snapshots" part. Yes, directly syncing a .git directory seems like a recipe for disaster with how often I've seen individual files lagging to sync, but I guess with BTRFS snaphots one can ensure that only a consistent view of a git directory is being backed up and synced.

Nah I truly do it the wrong way around. Syncthing on the git repos. And one of my device in the Syncthing cluster does btrfs snapshots minutely for recovery and further backups.

Because it's at a personal scale, the only time I can corrupt a git repo is if I work on the same repo (and it's workdir) from more than one device in the time it takes for Syncthing to replicate the changes.

But even then it's not a big deal because git fsck is quick. And I have my snapshots, and the syncthing versioning, and git defaults to two weeks before pruning. And because of how git works, using hash to identify contents, files are not easily overwritten either.

In 10y I only had one git corruption (I ran a command on the same repo on a different machine via ssh, yielding a synctning conflict). Syncthing kept copies of the conflict file. One commit disappeared from the history but not from the database. It was easy to rebase the changes. I think I used git fsck to deleted the syncthing versioned files.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#75

Post by School-Cotton »

blharr wrote: post #68
cvoss wrote: post #47

Earlier quoted context omitted.

You shouldn't use a word that can carry a precise mathematical meaning in a sentence that literally uses mathematical notation in order to speak precisely and then expect readers not to interpret the word in the precise mathematical way.

I somewhat agree, but for lack of a better word, what would you use? Quadratically doesn't have the same punch

“Dramatically” ?

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#76

Post by IshKebab »

plorkyeran wrote: post #13
SomaticPirate wrote: post #11

How was the flame graph created? (Not very familiar with C and the performance tools around it)

https://github.com/brendangregg/FlameGraph You record performance data with `perf`, then use the scripts there to turn it into a SVG.

I strongly recommend not using this. Instead use pprof - it has a MUCH better interactive flamegraph, plus other nice performance visualisations (e.g. a call graph):

https://github.com/google/pprof

    go install github.com/google/pprof@latest
    pprof -http=: prof.out
I normally collect the profiles with gperftools (https://github.com/gperftools/gperftools) and then just

    LD_PRELOAD=/usr/lib/libtcmalloc_and_profiler.so CPUPROFILE=prof.out
I've been meaning to try Samply though. Not sure if it works with pprof.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#77

Post by pjmlp »

IshKebab wrote: post #71
pjmlp wrote: post #39

A very good example that writing code in C doesn't help for performance, when the algorithms or data structures aren't properly taken into consideration.

I would say C makes this sort of thing far more likely because it's usually a ton of effort to obtain suitable containers. In C++ or Rust they have plenty of things like `unordered_set`/`HashSet` built in, so people are much more likely to use it and not go "eh, I'll use a for loop". In this case Git already had a string set, but it's still not standard so there's a good chance the original author just didn't know ab…

Yeah, not something that WG14 will ever care about.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#78

Post by echelon »

SAI_Peregrinus wrote: post #53
LgWoodenBadger wrote: post #18

IME, it has always turned out to be the correct decision to eliminate any n^2 operation in anything I’ve written. I don’t write exotic algorithms, but it’s always astounding how small n needs to be to become observably problematic.

I'd say the exception is when `n` is under about 10, and is counting some sort of hardware constrained thing (e.g. some operation over all CAN interfaces pesent on an OBDII connector can be O(n^(2)) since n will always be between 1 and 4). If you wouldn't have to physically replace hardware for `n` to increase, you really need to avoid n^2 operations. And even then consider them carefully, perhaps explicitly failing…

> perhaps explicitly failing if `n` gets too big

That's the problem. A lot of these quadratic time algorithms don't set limits.

Even 'n!' is fine for small 'n'. Real production use cases don't have small 'n'.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#80

Post by lassejansen »

echelon wrote: post #78
SAI_Peregrinus wrote: post #53

Earlier quoted context omitted.

I'd say the exception is when `n` is under about 10, and is counting some sort of hardware constrained thing (e.g. some operation over all CAN interfaces pesent on an OBDII connector can be O(n^(2)) since n will always be between 1 and 4). If you wouldn't have to physically replace hardware for `n` to increase, you really need to avoid n^2 operations. And even then consider them carefully, perhaps explicitly failing…

> perhaps explicitly failing if `n` gets too big That's the problem. A lot of these quadratic time algorithms don't set limits. Even 'n!' is fine for small 'n'. Real production use cases don't have small 'n'.

Or, phrased differently, if n has an upper limit, the algorithm is O(1).
Post reply on HN