One of the many reasons I moved to self hosting. I use ZFS to backup every 15 minutes, ... could do it even more frequently but that seems a little pointless. Also moved away from Gitlab because it's so damn slow.
ZFS is great, getting off gitlab would also be great, what did you switch to?
How we decreased GitLab repo backup times from 48 hours to 41 minutes
141–150 of 273 posts
Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes
#142One of the many reasons I moved to self hosting. I use ZFS to backup every 15 minutes, ... could do it even more frequently but that seems a little pointless. Also moved away from Gitlab because it's so damn slow.
This was my thought too, but I figured I just didn't understand the problem. Why use git commands to backup when a file system copy seems like it would do? zfs snapshot takes less than a second on any size repo. zfs send transfers just the changes since the last backup, as fast as your network, more or less.
Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes
#14348 hours is a crazy amount of time to spend just to compress a git folder, it's only a couple GB. 41 minutes still seems like quite a long time. Why aren't they just snapshotting and archiving the full git repo? Does `git bundle` add something over frequent ZFS backups?
zfs snapshots are difficult to offsite in non-zfs replicas, say like an S3 bucket. That said, there's another less known feature that bundles help out with when used with `git clone --bundle-uri` The client can specify a location to a bundle, or the server can send the client the bundle location in the clone results and the client can fetch the bundle, unpack it, and then update the delta via the git server, so it's…
Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes
#144IME, it has always turned out to be the correct decision to eliminate any n^2 operation in anything I’ve written. I don’t write exotic algorithms, but it’s always astounding how small n needs to be to become observably problematic.
My rule of thumb for 80%-90% of the problems is, if you need complicated algorithm, it means your data model isn't right. Sure, you do need complicated algorithms for compilers, db internals, route planning et all, but all things considered, those are minority of the use cases.
Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes
#145Are there any reimplementations of git, by professional programmers using real tools? The source in question — object.c — is "banging rocks together" material.
Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes
#146Earlier quoted context omitted.
ZFS is great, getting off gitlab would also be great, what did you switch to?
Gitea, i can't recommend it enough. Lightening fast compared to the proprietary offerings, clean simple UI like an older Github, and super simple to self host.
Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes
#147IME, it has always turned out to be the correct decision to eliminate any n^2 operation in anything I’ve written. I don’t write exotic algorithms, but it’s always astounding how small n needs to be to become observably problematic.
I spent a lot of time fixing n^2 in blink, but there were some fun surprises:
https://source.chromium.org/chromium/chromium/src/+/main:thi...
For large N without a cache :nth-child matching would be very slow doing n^2 scans of the siblings to compute the index. On the other hand for small sibling counts it turned out the cache overhead was noticably worse than just doing an n^2. (See the end of the linked comment).
This turns out to be true in a lot of surprising places, both where linear search beats constant time maps, and where n^2 is better than fancy algorithms to compensate.
Memory latency and instruction timing is the gotcha of many fun algorithms in the real world.
Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes
#148Earlier quoted context omitted.
[flagged]
This is why I said "We are discussing (mostly) tech here". I don't agree that creating a throwaway for every comment is "superior". It's basically spamming and it's even noted in the guidelines. Nazi Germany & Jews issue is different. There's an aspect of forcing, and this is unethical and wrong on so many levels, and I'll just leave the subject here. OTOH, from my perspective if you're afraid that you're writing a s…
It’s fine to be fearless. But don’t persecute someone for trying to protect themselves.
Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes
#149Re: How we decreased GitLab repo backup times from 48 hours to 41 minutes
#150Earlier quoted context omitted.
Fair, but `n log n` definitely is the historical "good enough to actually sleep at night" in my head, every time I see it I think of the prof who taught my first CSC course and our data structures course due to how often it came up. Also, the wise statement that 'memory is fairly cheap compared to CPU for scaling'. It's insane to see how often folks would rather manually open and scan a 'static-on-deploy' 20-100MB Js…
Not often but occasionally I will chose the nlogn algorithm which obviously has no bugs over the O(n) algorithm with no obvious bugs. Less brittleness is worth paying a few percent. Especially if it unmuddies the waters enough for someone to spot other accidental (time) complexity.
But I also don't dabble in this area nearly enough to know whether there's years of tears and toil finding out repeatedly that O(n) is ~impossible to implement and verify :)
| n | n log n |
| 5 | 8.0472 |
| 10 | 23.0259 |
| 25 | 80.4719 |
| 50 | 195.6012 |
| 100 | 460.5170 |