GitLab Database Incident – Live Report
docs.google.com
GitLab Database Incident – Live Report
1–10 of 621 posts
Re: GitLab Database Incident – Live Report
#2Not sure if the doc here is refreshing or scary. But Godspeed GitLab team. I've loved the product for about two years now, so curious to see how this plays out.
Re: GitLab Database Incident – Live Report
#3Are 'YP' the initials of an employee or is this an acronym I don't know?
Re: GitLab Database Incident – Live Report
#4Are 'YP' the initials of an employee or is this an acronym I don't know?
Yes, those are the initials of an employee here. Sorry for the confusion!
Re: GitLab Database Incident – Live Report
#5Are 'YP' the initials of an employee or is this an acronym I don't know?
I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.
Re: GitLab Database Incident – Live Report
#6Are 'YP' the initials of an employee or is this an acronym I don't know?
From looking at the context of the way YP is referenced (link to a slack archive), I believe YP is an employee.
Re: GitLab Database Incident – Live Report
#7"So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place."
Re: GitLab Database Incident – Live Report
#8Not sure if the doc here is refreshing or scary. But Godspeed GitLab team. I've loved the product for about two years now, so curious to see how this plays out.
It's both.
I very much appreciate their forthrightness and the way they conduct their company generally. Having said that, I have the code I work on, related content, and a number of clients on the service.
[edit for additional point]
They need the infrastructure guy they've been looking for sooner than later. I hope there's good progress on that front.
Re: GitLab Database Incident – Live Report
#923:00-ish
YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com
Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.