Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

551–560 of 621 posts

Re: GitLab Database Incident – Live Report

#551

Post by jamesmiller5 »

kentonv wrote: post #255
jamesmiller5 wrote: post #224

As I read the report I notice a lot of PostgreSQL "backup" systems depend on snapshotting from the FS & Rsync. This may work for database write logs, but it certainly will corrupt live git repositories that use local file system locking guarantees. NFS also requires special attention (a symlink lock) as writes can be acknowledged concurrently for byte offsets unless NFSv4 locking & compatible storage software is used…

> Git repo corruption from snapshotting tech (tarball, zfs, rsync, etc): The link discusses why rsync and tarballs are not good backup solutions. But, those wouldn't rightly be called "snapshots". "Snapshot" implies atomic, right? Surely an atomic snapshot would not corrupt git -- I would expect git, like any database, is designed to be recoverable after power failure, to which recovering from an atomic snapshot shou…

That is actually not the case.

Say a git server is in the middle of a write to refs/heads/master . You atomically snapshot the FS and then a power outage kills the server. The repository state will have a small chance of lock files that are never removed. Depending on the lock, future writes to a ref or to the repository can fail.

Not the worst situation as data lose won't occur but definitely not a stable state. If you treat git repositories as a service that needs a recovery step it would be fine, unfortunately most don't.

(edit) source: http://joeyh.name/blog/entry/difficulties_in_backing_up_live...

Re: GitLab Database Incident – Live Report

#552

Post by sbuttgereit »

Ajedi32 wrote: post #469
CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

Am I missing something? Where in this report are any individuals actually named? My understanding was that they're using initials in place of names specifically because they want to _avoid_ naming anyone.

The original versions of the document had names. Those were later replaced with initials.

I think the issue was in part that this document didn't appear to be a public "here's what's going on doc" as much as it was a doc they seemed to be using as a focal point for their own coordination efforts.

Re: GitLab Database Incident – Live Report

#553

Post by wolfgang42 »

luckystartup wrote: post #504

I have at least 10 private repos on GitLab, and many public ones. Even so, this is no big deal to me. That's the beauty of git. Even if all of their backups fail, I can just do a push and everything is back up there. I just hope my laptop doesn't die before they get it back online. EDIT: Was fun to put this little command together. Run this from your code directory, and it will push all of your gitlab repos. I'm goin…

From the incident report:

> Git repositories are NOT lost, we can recreate all of the projects whose user/group existed before the data loss, but we cannot restore any of these projects issues, etc.

Your fancy snippet will report that it has pushed no changes. The data that was lost was new issues, PRs, issue comments, and so on; I've never heard of anyone keeping backups of these on their local laptops.

Re: GitLab Database Incident – Live Report

#554

Post by why-el »

nickjj wrote: post #372

I wonder if this could have been avoided by using different subdomains. For example, instead of db1.cluster, what if it were named db-production.cluster. Would it have still happened? Probably not. I could totally see myself accidentally typing db1 out of muscle memory, but there's no way to accidentally type db-production.

That's what we do. `service-1-dev` and `service-1-prod`. Can't ssh to `prod` unless you are a really really qualified devops.

Re: GitLab Database Incident – Live Report

#555

Post by x0x0 »

orblivion wrote: post #485
x0x0 wrote: post #287

Earlier quoted context omitted.

if #2 is correct, holy shit did gitlab get lucky someone snapshotted 6 hours before. Dear you: it's not a backup until you've (1) backed up, (2) pushed to external media / s3; (3) redownloaded and verified the checksum; (4) restored back into a throwaway; (5) verified whatever is supposed to be there is, in fact, there, and (6) alerted if anything went wrong. Lots of people say this, and it's because the people sayin…

Do you have to download the entire backup or is a test backup using the same flow acceptable? I'm thinking about my personal backups, and I don't know if I have the time or space to try the full thing.

For plain files, pick your risk tolerance.

For DB backups, until you've actually loaded it back into the DB, recovered the tables, and tested a couple rows are bit identical to the source, it's a hope of a backup not a backup. Things like weird character set encodings can cause issues here.

Re: GitLab Database Incident – Live Report

#556

Post by wolfgang42 »

nickjj wrote: post #372

I wonder if this could have been avoided by using different subdomains. For example, instead of db1.cluster, what if it were named db-production.cluster. Would it have still happened? Probably not. I could totally see myself accidentally typing db1 out of muscle memory, but there's no way to accidentally type db-production.

They already do this. Both db1.cluster and db2.cluster are production machines; their staging equivalents are db1.staging and db2.staging. The confusion was between two production instances--one of which had the latest data, and the other was replicating. The intent was to delete the partially replicated data but the command was run on the current master database server instead.

Re: GitLab Database Incident – Live Report

#557

Post by djsumdog »

pmlnr wrote: post #442
arc_of_descent wrote: post #400

Don't prepare a backup plan. Prepare a recovery plan.

I agree with this. Prepare for destroyed (burnt-down level) machine, for datacentre failure, for stolen home server, for scratched blu-ray archives - in short, for the worst. And of course, hope for the best.

It's easier said than done in some companies where stake holders are always pushing for new stuff.

Unless those who fund what you're doing understand why disaster recovery is vital, you're going to see this.

Ideally you want devops in such a state that you create new lower environments that mirror production, complete with state/backup restoration, that's run automatically every week.

Re: GitLab Database Incident – Live Report

#558

Post by macintux »

bpchaps wrote: post #268
kelnos wrote: post #227

Earlier quoted context omitted.

You shouldn't let yourself get to that point. Your alerting system should alert you when disk is at 70% or something with a ton of margin. If it's not set up that way, go stop what you're doing and fix that. (Seriously.) If you're running your systems so they usually run at 90% disk usage, go give them more disk (or rotate logs sooner). And even assuming all that fails and I'm in that situation where I have seconds u…

If you lose a shoe while running across a highway, it's probably not worth the risk trying to get it back. The disk usage is just one example of many. Actions that you take really depends on what field you're in and, and again, legacy/inherited systems are completely filled with this sort of shit.. You can say that I shouldn't let get it to that point, but you're kinda dismissing the point I'm trying to make that - s…

I once tried to help a company with similar infrastructure insanity recover from a massive failure. Absolutely brutal.

When my team finally got services up and running (barely) after ~18 hours of non-stop work, the CTO demanded that we not go home and get some sleep until everything was exactly as it had been before the failure.

Not my happiest day.

Re: GitLab Database Incident – Live Report

#559

Post by danaliv »

> 2017/01/31 23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com

A long, frustrating day. Running destructive commands at 11pm. This is why pilots have duty time limits. YP should've been relieved by another engineer who was physically and mentally fresher. Human beings have limits, and when we reach them, we make more—and worse—mistakes. Any process that fails to account for this is broken.

Re: GitLab Database Incident – Live Report

#560

Post by luckystartup »

wolfgang42 wrote: post #553
luckystartup wrote: post #504

I have at least 10 private repos on GitLab, and many public ones. Even so, this is no big deal to me. That's the beauty of git. Even if all of their backups fail, I can just do a push and everything is back up there. I just hope my laptop doesn't die before they get it back online. EDIT: Was fun to put this little command together. Run this from your code directory, and it will push all of your gitlab repos. I'm goin…

From the incident report: > Git repositories are NOT lost, we can recreate all of the projects whose user/group existed before the data loss, but we cannot restore any of these projects issues, etc. Your fancy snippet will report that it has pushed no changes. The data that was lost was new issues, PRs, issue comments, and so on; I've never heard of anyone keeping backups of these on their local laptops.

> I've never heard of anyone keeping backups of these on their local laptops.

Hmm... That's an interesting idea!

You could do that on a separate (empty) branch. Maybe call it `__project`, and you could just have folders of markdown files. You could have two root folders for `issues/` and 'pull_requests/', and two subfolders in each for `./open/` and `./closed/`. And a simple command-line tool + web UI. You could just edit the file to add a comment.

It would be really nice to have a history and backup of all of your issues. I also like the fact that you could create or edit issues offline.

Then you could also set up a 2-way sync between your repo and GitLab / GitHub / Trello.

Post reply on HN