Live data from Hacker News

How we decreased GitLab repo backup times from 48 hours to 41 minutes

about.gitlab.com

171–180 of 273 posts

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#171

IME, it has always turned out to be the correct decision to eliminate any n^2 operation in anything I’ve written. I don’t write exotic algorithms, but it’s always astounding how small n needs to be to become observably problematic.

Good call. O(N^2) is the worst time complexity because it's fast enough to be instantaneous in all your testing, but slow enough to explode in prod. I've seen it several times before, and it's exactly what happened here.

[deleted]

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#173
post #6

"fixed it with an algorithmic change, reducing backup times exponentially" If the backup times were O(n^2), are they now O(n^2 / 2^n)? I would guess not.

This is not the precise mathematical definition of exponential, but rather the colloquial one, where it just means "a lot".

>Ultimately, we traced the issue to a 15-year-old Git function with O(N²) complexity and fixed it with an algorithmic change, reducing backup times exponentially.

No, not in the exact same sentence as a big-O. That's either author error, or an intentional troll. Either way it's egg on their faces.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#174
post #47

Earlier quoted context omitted.

This is not the precise mathematical definition of exponential, but rather the colloquial one, where it just means "a lot".

You shouldn't use a word that can carry a precise mathematical meaning in a sentence that literally uses mathematical notation in order to speak precisely and then expect readers not to interpret the word in the precise mathematical way.

Especially when the colloquial meaning derives from the mathematical meaning.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#175
post #165

Earlier quoted context omitted.

Absolutely not. If the cost of doing something goes above quadratic, you shouldn't do it at all. Because essentially every customer interaction costs you more than the one before. You will never be able to come up with ways to cover that cost faster than it ramps. You are digging a hole, filling it with cash and lighting it on fire. If you can't do something well you should consider not doing it at all. If you can on…

Chess engines faced worse than quadratic scaling and came out the other side… Software operates in a crazy number of different domains with wildly different constraints.

I believe hinkley was commenting on things that are quadratic in the number of users. It doesn't sound like a chess engine would have that property.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#176
post #47

Earlier quoted context omitted.

You shouldn't use a word that can carry a precise mathematical meaning in a sentence that literally uses mathematical notation in order to speak precisely and then expect readers not to interpret the word in the precise mathematical way.

You should if you expect your readers to be normal humans who understand obvious context, and not pedantic HN readers who understand obvious context but delight in nit-picking it anyway.

>pedantic

Who the fuck do you think is the intended audience for an article about an algorithm in `git bundle create`? I spent approximately two minutes of my life trying to figure out where the O(n^2) algorithm was being invoked in such a way that it influenced an exponential.

Exponential was bolded in the same sentence as a big-O. 50/50 troll/author oversight.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#177
post #68
post #47

Earlier quoted context omitted.

You shouldn't use a word that can carry a precise mathematical meaning in a sentence that literally uses mathematical notation in order to speak precisely and then expect readers not to interpret the word in the precise mathematical way.

I somewhat agree, but for lack of a better word, what would you use? Quadratically doesn't have the same punch

Algorithmic? Big-O? Polynomially? Linear improvement? O(n^2) to O(n)? Or if you want to be less mathematically precise: enormous improvement?

Using exponential in this way in any context is a faux pas, because it's highly ambiguous, and requires context for clarification. But in this situation the context clearly resolved to the mathematically accurate definition, except it was used in the other way.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#178
post #111
post #47

Earlier quoted context omitted.

You shouldn't use a word that can carry a precise mathematical meaning in a sentence that literally uses mathematical notation in order to speak precisely and then expect readers not to interpret the word in the precise mathematical way.

“Words mean things.” If you can’t agree with this, then you shouldn’t be speaking or writing, IMO. Those who argue that words that mean different things are actually equivalent have no business dealing with language.

I understood every word, phrase, and sentence you wrote. But I did not understand your point. Still, I got the meaning of your words, so presumably you're satisfied.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#179
post #124

Earlier quoted context omitted.

That is not true unless n^C / e^n = log(n) where C is some constant, which it is not. The difference between log(n) and some polynomial is logarithmic, not exponential.

It depends if you consider "speedup" to mean dividing the runtime or applying a function to the runtime. I.e. you are saying and f(n) speedup means T(n)/f(n), but others would say it means f(T(n)) or some variation of that.

The man, or llm, used the mathematically imprecise definition of exponential in a sentence with a big-O notation. I don't think he's going to be writing entire arguments formally.

Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes

#180
post #6

"fixed it with an algorithmic change, reducing backup times exponentially" If the backup times were O(n^2), are they now O(n^2 / 2^n)? I would guess not.

Meaningless and non-constructive pedantry.

>pedantry

I wasted 2 minutes of my life looking for the exponential reduction. So did many others.

Now I'm wasting more of my life shit posting about it, but at least that's a conscious choice.

Post reply on HN