My team and I switched from bitbucket to gitlab a few months ago and we love the transition. Gitlab provides a lot of value to me and my team who are learning how to code while working on side projects. Although we cannot send merge requests today because of this issue, we are all cheering them on. I’m very happy that they are so transparent about their issues because my team and I are learning so much from their report and insights here on HN comments. Good luck!
GitLab Database Incident – Live Report
191–200 of 621 posts
Re: GitLab Database Incident – Live Report
#192Earlier quoted context omitted.
'db1' vs. 'db2' is still insufficiently clear, though. Even better would be e.g. to name development systems after planets and production systems after superheroes. Very few people would mistake 'superman' for either 'green-lantern' or 'pluto,' but it's really easy to mistake 'sfnypudb13' for 'sfnydudb13.'
Or skip the middleman: prod-db1, prod-db2, dev-db1, dev-db2.
Visually there's not a whole lot of difference. Ideally, you want something where the shape of each name is reasonably distinct from all the rest — otherwise folks will just ignore that odd blot before the prompt.
And then, once you've started naming things as $ENV-$TYPE, someone will want to cram in the location, and the OS, and the team which maintains it, and the customer, and and and. Then someone will reduce all of those identifiers into single characters … and you'll be in the situation I mentioned. Clearly clpudb1 is CharlesCorp's first London production database system!
Re: GitLab Database Incident – Live Report
#193Earlier quoted context omitted.
That is why we went with initials. And I hope customers understand the blame is with all of us, starting with me. Not with the person in the arena. https://twitter.com/sytses/status/826598260831842308
It seems to me that, as a customer, it is blame-shifting away from the company to a particular person. Blameless post-mortems are great, but when speaking to people outside the company I think it is important to own it collectively, "after a second or two we notice we ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com." I believe this isn't your intention, but that is how I interpreted it.
In our postmortems we explicitly avoid referring to names and only refer to "engineers" or specific teams. There is no reason to refer to specific names if your intention is a systems/process fix.
Re: GitLab Database Incident – Live Report
#194> 2017/01/31 23:00-ish > YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. > After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com > 2017/01/31 23:27 > YP terminates the removal, but it’s too late. Of around 310 GB only about 4.5 GB is left The naming couldn't be more confusing
In what way? They have two production database servers, db1 and db2.
Re: GitLab Database Incident – Live Report
#195You don't have a production ready backup system until you've used it.
Re: GitLab Database Incident – Live Report
#196Earlier quoted context omitted.
BBC's Horizon has a really good episode about checklists and how they're used to prevent mistakes in hospitals, and how they're being adopted in other environments in light of that success. It's called How To Avoid Mistakes In Surgery for the interested.
There are very, very few situations in life where there's not enough time to take a time-out, sitrep, or checklist. I work in EMS when not in IT, and even bringing a trauma or cardiac arrest patient into the Emergency Room, there is still time to review and consider. If there's time in Emergency Medicine, there's time in IT.
I worked in software to help manage this for a while. There are still checklists but they are produced ahead of schedule. Every instrument a nurse takes off the tray is counted and then checked at the end for instance.
Re: GitLab Database Incident – Live Report
#197"Of around 310 GB only about 4.5 GB is left" What is gitlab storing in their database? From what I understand, the repos were untouched by the DB problems, so what is taking up a third of a terabyte of DB space?
Tickets, tags, labels, users information/data and more. Sounds like they also host snippets out of the PostgreSQL database.
310 GB for a database is not all that large.
Re: GitLab Database Incident – Live Report
#198Earlier quoted context omitted.
Yes, good tip from "Turn the Ship Around" by David Marquet is to use the "I intend to" model. For every action you are going to undertake that is critical, first announce your intentions and give enough time for reactions from others before following through.
I saw that Space Shuttle landing video that was kicking around recently. In that they also had explicit "I agree" responses to any observation like "You're a bit below flight path". Quick, positive acknowledgment of anomalous events or deviations. Seemed really ... sane.
do you happen to have the link for that video (and related discussion if posted on HN)?
Re: GitLab Database Incident – Live Report
#199"So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place."
Was this document meant to be public ? It this is the case, the honesty is perhaps a bit too radical.
They tweeted a link to that document on their status account. It is unusually candid.
Re: GitLab Database Incident – Live Report
#200Earlier quoted context omitted.
Or skip the middleman: prod-db1, prod-db2, dev-db1, dev-db2.
Visually there's not a whole lot of difference. Ideally, you want something where the shape of each name is reasonably distinct from all the rest — otherwise folks will just ignore that odd blot before the prompt. And then, once you've started naming things as $ENV-$TYPE, someone will want to cram in the location, and the OS, and the team which maintains it, and the customer, and and and. Then someone will reduce all…
I strongly argue against those types of abbreviations in naming, for exactly the points you make.
Here's how I view it:
The distinction between prod and dev is pretty clear cut. The words "dev" and "prod" have significantly different shapes and are immediately unambiguous. There's no need to remember a superhero vs astronomical object distinction.
As the number of database server instances grows, you've got another level of naming issues that arises when needing to distinguish between hosts and db server instances—and perhaps even database/schema independence if necessary. Staying consistent and easy-to-use with an arbitrary naming scheme becomes increasingly unwieldy, in my opinion.