Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we decreased GitLab repo backup times from 48 hours to 41 minutes

about.gitlab.com

Post reply on HN
Print view View original

31–40 of 273 posts

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#31

Post by haroldp »

tomxor wrote: post #16

One of the many reasons I moved to self hosting. I use ZFS to backup every 15 minutes, ... could do it even more frequently but that seems a little pointless. Also moved away from Gitlab because it's so damn slow.

This was my thought too, but I figured I just didn't understand the problem. Why use git commands to backup when a file system copy seems like it would do? zfs snapshot takes less than a second on any size repo. zfs send transfers just the changes since the last backup, as fast as your network, more or less.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#32

Post by DamonHD »

nayak wrote: post #29
DamonHD wrote: post #26

Here comes an unpopular nitpick: "... we traced the issue to a 15-year-old Git function with O(N²) complexity and fixed it with an algorithmic change, reducing backup times exponentially." Uh no you didn't. Not possible. At most a polynomial reduction is possible else complexity theory needs a re-write. (OK, yes, k could be doing some heavy lifting here, but I doubt it.) If you are going to quote a maths formula then…

OP here. Feedback is always welcome, I did mean exponentially in the colloquial sense. I do see how it is confusing here, will change it.

Thank you.

(I don't think that anyone should use "exponentially" that way: it is an art term with a specific and particular meaning, so find another word if you mean something else! Like misusing specific legal or sporting terms...)

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#33

Post by [deleted] »

linguistbreaker wrote: post #25
chrisweekly wrote: post #21

Earlier quoted context omitted.

I'm not the OP you're responding to, but to be fair, in a sentence about big-O perf characteristics, which includes the word "algorithms", using "exponentially" in a colloquial non-technical sense is an absolutely terrible word choice.

Exponentially bad word choice even... since we're using that word however we want now? I don't think this is meaningless or non-constructive pedantry - we're a technical community and those are technical words.

[deleted]

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#34

Post by unsnap_biceps »

Arcuru wrote: post #23

48 hours is a crazy amount of time to spend just to compress a git folder, it's only a couple GB. 41 minutes still seems like quite a long time. Why aren't they just snapshotting and archiving the full git repo? Does `git bundle` add something over frequent ZFS backups?

zfs snapshots are difficult to offsite in non-zfs replicas, say like an S3 bucket.

That said, there's another less known feature that bundles help out with when used with `git clone --bundle-uri` The client can specify a location to a bundle, or the server can send the client the bundle location in the clone results and the client can fetch the bundle, unpack it, and then update the delta via the git server, so it's a lot lighter weight on the server for cloning large repos, and a ton faster for the client for initial clones.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#35

Post by Lammy »

> What this means for GitLab customers — [a bunch of stuff about how customers can now back up more frequently and more robustly]

Realtalk: They should rewrite this post's headline to be in a positive tense instead of leading with a negative word. I'm glad I read the post, because it is a cool and good fix, but I saw “Decreasing […] repo backup” and my first thought was that it was an announcement of some service downgrade like some sort of cost-cutting measure.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#36

Post by bombela »

DamonHD wrote: post #26

Here comes an unpopular nitpick: "... we traced the issue to a 15-year-old Git function with O(N²) complexity and fixed it with an algorithmic change, reducing backup times exponentially." Uh no you didn't. Not possible. At most a polynomial reduction is possible else complexity theory needs a re-write. (OK, yes, k could be doing some heavy lifting here, but I doubt it.) If you are going to quote a maths formula then…

I can forgive exponential as a figure of speech. You missed when they describe using a map for the side effect of [key] dedupliction.

Instead of saying "set". The code itself uses the type "strset".

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#37

Post by ctxc »

DamonHD wrote: post #32
nayak wrote: post #29

Earlier quoted context omitted.

OP here. Feedback is always welcome, I did mean exponentially in the colloquial sense. I do see how it is confusing here, will change it.

Thank you. (I don't think that anyone should use "exponentially" that way: it is an art term with a specific and particular meaning, so find another word if you mean something else! Like misusing specific legal or sporting terms...)

Art term?
Post reply on HN