Live data from Hacker News

How we decreased GitLab repo backup times from 48 hours to 41 minutes

about.gitlab.com

131–140 of 273 posts

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#131
TLDR if you only add objects to an array that are already not contained in said array you won't have to iterate back through it to remove the duplicates you created.

wild that this a pattern like this would be part of git-core, but I guess we all overlook stuff on a regular basis

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#132

Earlier quoted context omitted.

Bruce Dawson says: I like to call this Dawson’s first law of computing: O(n^2) is the sweet spot of badly scaling algorithms: fast enough to make it into production, but slow enough to make things fall down once it gets there. https://bsky.app/profile/randomascii.bsky.social/post/3lk4c6...

The second law is that O(n * log n) is for practical intents and purposes O(n).

Skiena has a great table in his algorithms book mapping time complexity to hypothetical times for different input sizes.

For n of 10^9, where lgn takes 0.03 us and n takes 1 s, nlgn takes 29.9 s and n^2 takes 31.7 years.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#133

Earlier quoted context omitted.

[flagged]

You don't have to enter any e-mail address to get an HN account. You login from a (Firefox) incognito window and get your cookies deleted the moment the window is closed. Why you're so afraid to let your ideas and views collect under a single account? Are they that controversial or are you weary of your own thoughts and don't want to see them again, or are you afraid to own your views as yours? We're talking (mostly)…

[flagged]

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#134
post #6

"fixed it with an algorithmic change, reducing backup times exponentially" If the backup times were O(n^2), are they now O(n^2 / 2^n)? I would guess not.

The algorithm complexity went down in the function they patched (6x improvement in their benchmark), but in the context of how they benefited with how they were using the algorithm the impact was much larger (improved to taking 1% of the time), which is plausibly exponential (and figuring out the actual complexity is neither relevant nor an economic use of time).

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#135
post #124
post #62

Earlier quoted context omitted.

If you replace an n^2 algorithm with a log(n) lookup you get an exponential speed up. Although a hashmap lookup is usually O(1), which is even faster.

That is not true unless n^C / e^n = log(n) where C is some constant, which it is not. The difference between log(n) and some polynomial is logarithmic, not exponential.

But if you happen to have n=2^c, then an algorithm with logarithmic complexity only needs c time. Thats why this is usually referred to as exponential speedup in complexity theory, just like from O(2^n) to O(n). More concretely if the first algorithm needs 1024 seconds, the second one will need only 10 seconds in both cases, so I think it makes sense.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#136
post #6

"fixed it with an algorithmic change, reducing backup times exponentially" If the backup times were O(n^2), are they now O(n^2 / 2^n)? I would guess not.

I interpreted that as n->log(n) since log and exp are inverses.

Also because I've often heard tha the quantum Fourier transform is an exponential speedup over the discrete Fourier transform, and there the scaling goes n^2->nlogn.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#137

Earlier quoted context omitted.

You don't have to enter any e-mail address to get an HN account. You login from a (Firefox) incognito window and get your cookies deleted the moment the window is closed. Why you're so afraid to let your ideas and views collect under a single account? Are they that controversial or are you weary of your own thoughts and don't want to see them again, or are you afraid to own your views as yours? We're talking (mostly)…

[flagged]

This is why I said "We are discussing (mostly) tech here". I don't agree that creating a throwaway for every comment is "superior". It's basically spamming and it's even noted in the guidelines.

Nazi Germany & Jews issue is different. There's an aspect of forcing, and this is unethical and wrong on so many levels, and I'll just leave the subject here.

OTOH, from my perspective if you're afraid that you're writing a sensitive comment, you can create a throwaway. That's justifiable IMHO, but creating three accounts to discuss maps vs. loops, now that's different.

If we're talking about being ridiculous, bringing up Nazi Germany vs. Jews issue to a technical discussion is more ridiculous than the alleged ridiculousness of me asking the OP about their fears. To close, my questions was not to belittle or shame the OP, they were genuine. I'm not that person who jabs for giggles.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#138
post #124
post #62

Earlier quoted context omitted.

If you replace an n^2 algorithm with a log(n) lookup you get an exponential speed up. Although a hashmap lookup is usually O(1), which is even faster.

That is not true unless n^C / e^n = log(n) where C is some constant, which it is not. The difference between log(n) and some polynomial is logarithmic, not exponential.

It depends if you consider "speedup" to mean dividing the runtime or applying a function to the runtime.

I.e. you are saying and f(n) speedup means T(n)/f(n), but others would say it means f(T(n)) or some variation of that.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#140

Cool discovery but the article could have been about 1/10 as long and still communicated effectively. At least they didn't post it as a video, so it was easy to skim to the important details.

For those that haven't read the article yet, scroll down to the flame graph and start reading unit it starts talking about back porting the fix. Then stop.
Post reply on HN