Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

471–480 of 621 posts

Re: GitLab Database Incident – Live Report

#471

Post by riebschlager »

louiz wrote: post #445
riebschlager wrote: post #221

I'm another happy GitLab user, but things like this always kinda freak me out. Do any of you use any repo-mirroring strategy? Something a little more automated than pushing to and maintaining separate remotes? For example, would it be worth it to spin up a self-hosted GitLab instance and then script nightly pulls from GitLab.com? Edit: Answered my own question! If anyone else was curious: http://stackoverflow.com/que…

Why would you use gitlab.com if you have a self-hosted Gitlab instance?

Because I trust(ed) GitLab's backup procedures more than my own.

Re: GitLab Database Incident – Live Report

#472

Post by marcinzm »

omgdresser wrote: post #147
overcast wrote: post #74

Earlier quoted context omitted.

Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be con…

I went through all of the insane interviews except the CEO interview for the production engineer position and was rejected. I brought quite a bit of high-level operational experience from a few different places, including some pretty high-traffic video sites, HPC and a lucrative e-commerce site. The reason for my rejection? I had an ongoing side project and development/consulting gig which had been paying the bills f…

>I guess they weren't risk-averse enough in their operations though, so I'm glad I didn't get through. It sounds like it would have been an uphill battle all the way to make changes to keep things sane.

I think they were risk averse which is the problem except they were judging against the wrong candidate profile. The risk adverseness meant they never hired anyone who diverged from their incorrect profile.

Re: GitLab Database Incident – Live Report

#473

Post by jschulenklopper »

ausjke wrote: post #470

That's why I do not use github/gitlab/whatever to host the part of my code that is too critical to me. I push it to my ssh/git server and use local UI to interact with it instead. Sometimes source code is very valuable and you just can not make any mistakes with it.

> I do not use github/gitlab/whatever to host the part of my code that is too critical to me

In this specific case, GitLab mentioned that code repositories are fine. It's the database part with issues and pull requests that they are restoring.

https://twitter.com/gitlabstatus/status/826662763577618432

Re: GitLab Database Incident – Live Report

#474

Post by marcinzm »

developer2 wrote: post #246
overcast wrote: post #74

Earlier quoted context omitted.

Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be con…

>> candidates will be invited to schedule a first 45 minute behavioral interview with the Infrastructure Lead Yes, go right ahead and filter out some (disclaimer before the rant: some, not all) of the best talent. The kind of potential employee that gets rejected due to perceived personality problems is exactly the kind of person who would tell management to shove a stick up their ass for demanding a 2 week deadline…

As others have said, abrasiveness isn't a virtue especially in a role that requires others to change their behavior. There are in fact diplomatic ways to convey the same information which have a much higher chance of causing others to listen and getting everyone to adopt new processes. That someone is diplomatic and knows/care about their stuff are not mutually exclusive.

That said, depending on their interview process they may be selecting out candidates who even diplomatically disagree about the existing ops processes which results in less than stellar hires.

Re: GitLab Database Incident – Live Report

#475

Post by mdekkers »

YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com

My heart honestly goes out to YP. This is a terrible sysops _oh shit_ moment.

Re: GitLab Database Incident – Live Report

#476

Post by YorickPeterse »

CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.

Re: GitLab Database Incident – Live Report

#477

Post by khana »

nanch wrote: post #413
CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the mo…

Incredible.

Re: GitLab Database Incident – Live Report

#480

Post by jschulenklopper »

musicmatze wrote: post #318

They say that git data (repos and wikis) are not affected... well ... if they would have had their PRs and Issues in git repositories, too... Disclaimer: Worked on a POC for exactly this last semester and going to publish my results in the next few weeks.

> if they would have had their PRs and Issues in git repositories, too

Well, perhaps a git repository isn't the right persistency structure for non-code items like issues and pull requests. That a git repository is a good structure for document-like entities (like code files) and tracking changes/versions, doesn't mean that it's a good choice for highly-relational or highly-dynamic objects.

Post reply on HN