451–460 of 621 posts
The recovery process seems very slow with ~50mbit/sec. Could that be an issue related to cloud providers? I heard that issue quite often when dealing with AWS/Azure. Even HDDs should have much higher throughput for that kind of transfer. If they had dedicated hardware in 2 datacentres on the same continent, copying between those servers should easily be possible at 250mbit/s or more (from my experience). Especially a…
Earlier quoted context omitted.
This sounds good until you consider that many systems are utilizing multiple drives. When someone is expecting to delete a large file and it ends up on a different drive, problems could arise.
Renaming a file should not move its data, right? So rename the file into .file.$(date -Iseconds).trash (but make sure that no legitimate files are ever named in this pattern). Then put that file path into a global /var/trashlist. To cleanup, you just check that file for expired trash and make the final deletion.
Proposed tweaks: symbolic link into /var/trashlist directory, where the name of the symbolic link is "--". Timestamp first so we can stop once we hit the first too-recent timestamp, random stub to unique the original base name if two different files In different directories are deleted at the same timestamp, original file name for inspection.
Earlier quoted context omitted.
Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the mo…
In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)
When people answer that question honestly and with humility it is a big plus.
As I read the report I notice a lot of PostgreSQL "backup" systems depend on snapshotting from the FS & Rsync. This may work for database write logs, but it certainly will corrupt live git repositories that use local file system locking guarantees. NFS also requires special attention (a symlink lock) as writes can be acknowledged concurrently for byte offsets unless NFSv4 locking & compatible storage software is used…
The other good alternative is atomic backups (like with pg_dump), but that does put some extra load on your database that may be unacceptable.
Yes, you will want something different for your git repositories. There is no backup procedure that is best for all cases.
Earlier quoted context omitted.
I bet it writes to a log file. It just doesn't alert anyone on failure so the log just grows and grows daily with the same error.
Or it alerts people, but on the same channel every other piece of infrastructure alerts them, and they have a severe case of false positives. I've seen that many more times than I've seen the "no alert" option.
Existing team: "Yah don't worry about that. It does that every day. We'll get to it sometime soon."
Earlier quoted context omitted.
Also, as a safety net, sometimes you don't need to run `rm -rf` (a command which should always be prefaced with 5 minutes of contemplation on a production system). In this case, `rmdir` would have been much safer, as it errors on non-empty directories.
These days, I've been very implicit in how I run rm. To the extent that I don't do rm -rf or rmdir (edit: immediately), but in separate lines as something like: pushd dir ; find . -type f -ls | less ; find . -type f -exec rm '{}' \; ; popd ; rm -rf dir It takes a lot longer to do, but I've seen and made enough mistakes over the years that the forced extra time spent feels necessary. It's worked pretty well so far --…
Earlier quoted context omitted.
These days, I've been very implicit in how I run rm. To the extent that I don't do rm -rf or rmdir (edit: immediately), but in separate lines as something like: pushd dir ; find . -type f -ls | less ; find . -type f -exec rm '{}' \; ; popd ; rm -rf dir It takes a lot longer to do, but I've seen and made enough mistakes over the years that the forced extra time spent feels necessary. It's worked pretty well so far --…
BTW, find ... -delete avoids any potential shell escaping weirdness and saves you a fork() per file.
Very interesting article: https://www.dwheeler.com/essays/fixing-unix-linux-filenames.....
As I read the report I notice a lot of PostgreSQL "backup" systems depend on snapshotting from the FS & Rsync. This may work for database write logs, but it certainly will corrupt live git repositories that use local file system locking guarantees. NFS also requires special attention (a symlink lock) as writes can be acknowledged concurrently for byte offsets unless NFSv4 locking & compatible storage software is used…
I see LVM[1] mentioned in the notes. It allows you to, among other things, snapshot a filesystem atomically which you could then mount read-only to a separate location to read for backups or export to a different environment. That would give you a point in time view of the state of all the repos that should be as consistent as a "stop the world then backup" approach. [1]: https://en.wikipedia.org/wiki/Logical_volume_…
It's definitely not as consistent as "stop the world then backup" because the filesystem is dirty, and the database is dirty. It's equivalent to yanking the power cord from the back of the system, then running fsck, then replaying all the uncommitted transactions from the WAL.
It's for this reason that I use ZFS for snapshotting. It guarantees filesystem consistency and data consistency at a given point in time. It'll still need to deal with replaying the WAL, but you don't need to worry about the filesytem being unmountable (it does happen), and you don't need to worry about the snapshot becoming unreadable (once the snapshot LV runs out of space). LVM was neat in the early 2000s, but there are much better solutions today.
Earlier quoted context omitted.
It seems to me that, as a customer, it is blame-shifting away from the company to a particular person. Blameless post-mortems are great, but when speaking to people outside the company I think it is important to own it collectively, "after a second or two we notice we ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com." I believe this isn't your intention, but that is how I interpreted it.
In our postmortems we explicitly avoid referring to names and only refer to "engineers" or specific teams. There is no reason to refer to specific names if your intention is a systems/process fix.
You write CEOs name on all your publications, of course always taking credit/glory, but why not let engineers do the same, take credit/ownership when doing a nice commits, and when fucking up. We're all people first, and prefer to speak/talk to people and not Engineering Team MailBox at Enterprise Corporation.