Live data from Hacker News

How we decreased GitLab repo backup times from 48 hours to 41 minutes

about.gitlab.com

91–100 of 273 posts

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#91

Earlier quoted context omitted.

If the requirement is to check uniqueness, what assumptions could possibly cause a bug? In this case, why does it matter if the uniqueness is tested with a nested for loop or with a map? There are many identical ways to check uniqueness, some being faster than others.

[flagged]

Why are you making a new account for each comment? You seem to be deliberately avoiding HN's moderation system

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#92
post #35

> What this means for GitLab customers — [a bunch of stuff about how customers can now back up more frequently and more robustly] Realtalk: They should rewrite this post's headline to be in a positive tense instead of leading with a negative word. I'm glad I read the post, because it is a cool and good fix, but I saw “Decreasing […] repo backup” and my first thought was that it was an announcement of some service dow…

I don’t think it’s unreasonable to expect interested people to read five words from the title. I know people don’t always do that, but complaining about it as if they did something wrong is ridiculous.

No one was complaining.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#93
post #28

Earlier quoted context omitted.

I disagree. Misuse of the word "exponential" is a major pet peeve of mine. It's a particular case of the much more common "use mathematically precise phrasing to sound careful/precise" that you often find in less than honest writing. Here they are actually using it to refer to growth functions (which is rare for this error) and being honest (which is also rare IMO) but it's still wrong. They should have written about…

Sloppy writing is up orders of magnitude lately.

It will decimate readership.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#94
post #23

48 hours is a crazy amount of time to spend just to compress a git folder, it's only a couple GB. 41 minutes still seems like quite a long time. Why aren't they just snapshotting and archiving the full git repo? Does `git bundle` add something over frequent ZFS backups?

... so they added caching to things that should have been cached?

... is this really the way people "back up" git repos? I mean, it is git, so isn't there some way to mirror changes to the repo in another repo and just use ZFS / snapshots / backup software / etc to do that? It's a distributed version control system. Just make sure the version control information is ... distributed?

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#95

Cool discovery but the article could have been about 1/10 as long and still communicated effectively. At least they didn't post it as a video, so it was easy to skim to the important details.

Yes. I read the whole article thinking that this must have been generated by LLM, because at least the style remembers it.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#96
post #32
post #29

Earlier quoted context omitted.

OP here. Feedback is always welcome, I did mean exponentially in the colloquial sense. I do see how it is confusing here, will change it.

Thank you. (I don't think that anyone should use "exponentially" that way: it is an art term with a specific and particular meaning, so find another word if you mean something else! Like misusing specific legal or sporting terms...)

Which term is appropriate here? What would you suggest? (honest question)

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#97
post #50

I'm confused why you wouldn't simply snapshot the block-level device if the protocol of the information on top is going to cause this much headache. Quiescing git operations for block level activity is probably not trivial, but it sounds like an easier problem to solve to me. This is the approach I've taken with SQLite in production environments. Turn on WAL and the problem gets even easier to solve. Customer configu…

> Git presumably doesn't have something approximating a WAL, so I understand the hesitation with this path.

Bingo. One of the worst problems is helping a client piece back together a corrupted repo when they are using snapshots. Check my profile to see how I know. :)

It's usually an OMG down scenario, and then you are adding in the "oh no, now the restore is corrupted."

It's fixable but it's definitely annoying.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#100

Earlier quoted context omitted.

The second law is that O(n * log n) is for practical intents and purposes O(n).

To be clear though, that isn't his second law, at least as of two months ago, according to https://bsky.app/profile/randomascii.bsky.social/post/3lk4c6...

Fair, but `n log n` definitely is the historical "good enough to actually sleep at night" in my head, every time I see it I think of the prof who taught my first CSC course and our data structures course due to how often it came up.

Also, the wise statement that 'memory is fairly cheap compared to CPU for scaling'. It's insane to see how often folks would rather manually open and scan a 'static-on-deploy' 20-100MB Json file for each request vs just parsing it into structures in memory (where, for most cases, the in memory usage is a fraction of the json itself) and just caching the parsed structure for the length of the application.

Post reply on HN